🤝 TCP in Depth
A deep dive into the protocol that carries most of the internet: the connection state machine, sequencing and SACK, loss detection and timers, flow and congestion control, the bandwidth-delay product, options, and how TCP meets a modern NIC.
7.1 The reliable-stream illusion
IP gives hosts a way to send datagrams across an internetwork. That word, datagram, is important: each IP packet is an independent attempt. Routers may drop it when queues overflow. Different packets may take different paths and arrive out of order. A packet may be duplicated by a lower layer or by a confused retransmission system below IP. A packet may survive, but arrive so late that the receiver has already moved on. IP checks enough structure to route and deliver a packet to the next protocol, but it does not remember a conversation and it does not promise that packet 17 arrives after packet 16, or arrives at all.
TCP turns that hostile service into a useful programming abstraction: an ordered, reliable, full-duplex byte stream between two endpoints. The application writes bytes into a socket. The peer reads the same bytes in the same order, unless the connection fails and the application is told that the stream can no longer make progress. That is the reliable-stream illusion. It is endpoint machinery: sequence numbers, acknowledgments, retransmission timers, receive queues, send queues, advertised windows, congestion windows, and a state machine that both sides maintain consistently enough to hide packet-level disorder.
TCP is better understood as a protocol for managing shared state than as a protocol for shipping packets. A TCP segment is merely one carrier for part of the byte stream plus control information. The stream is indexed by byte sequence numbers. If an application writes 100 KiB, the sender may emit many MSS-sized segments, one large offloaded buffer that the NIC later segments, or a sequence of smaller writes that the stack coalesces. The receiver does not owe the application the same boundaries. Two send() calls can be observed by one recv(), and one send() can be split across many recv() calls. TCP preserves byte order and byte contents; it does not preserve application message framing.
That distinction is a common source of bugs in low-level C code. If a protocol says "read a 4-byte length, then read that many bytes", the code must loop until those bytes have arrived. It cannot assume that one recv() corresponds to one peer send(), one Ethernet frame, one IP packet, or one TCP segment. The API exposes a stream because TCP's contract is a stream contract.
A TCP connection is named, at the transport layer, by the four values that identify the two endpoints: local IP address, local TCP port, remote IP address, and remote TCP port. In packet dumps this is the familiar four-tuple. It lets a host demultiplex incoming segments to the correct socket even when thousands of connections share the same local port on a server. A listening socket on 0.0.0.0:443 or [::]:443 is not itself one connected stream; accepted sockets become distinct connections once the peer address and peer port are known. For a NIC or driver engineer, this tuple also appears in receive-side scaling, flow hashing, filtering rules, and steering decisions.
The chapter unpacks four jobs that create the illusion.
- Connection setup and teardown establish which endpoints are talking, synchronize initial sequence numbers, negotiate options, and move both endpoints through well-defined states. TCP is full-duplex, so each direction of the byte stream has its own sequence space and can close independently.
- Reliability is built from sequence numbers, cumulative acknowledgments, selective acknowledgments where negotiated, and retransmission. The sender keeps bytes until it has evidence that the peer TCP has accepted responsibility for them. The receiver uses sequence numbers to place data into the stream, discard duplicates, and hold later bytes while an earlier gap is missing.
- Flow control protects the receiver. Every TCP receiver has finite memory and finite application drain rate. The advertised receive window tells the sender how many more bytes may be outstanding beyond the acknowledged stream position. If the application stops reading, the window shrinks, and the peer must stop sending new data even if the path has spare capacity.
- Congestion control protects the network. A sender also maintains a limit based on inferred path capacity and congestion signals such as loss or explicit congestion notification. Classic TCP uses slow start, congestion avoidance, fast retransmit, and fast recovery; modern stacks may use CUBIC or BBR, but the central idea remains that a TCP endpoint must not treat the Internet as an infinite sink.
The illusion has sharp edges. TCP can deliver bytes reliably and in order only while the connection survives. It cannot tell the application whether the peer process parsed a request, committed a transaction, or flushed data to disk. An acknowledgment means the peer TCP stack has taken responsibility for a byte range, not that the remote application has acted on it. TCP ordering also causes head-of-line blocking: if byte N is missing, later bytes may already be in the receive queue, but the stream API cannot deliver them before N.
For systems and NIC work, preserving this abstraction is the rule behind details that otherwise look like performance tricks. TCP segmentation offload lets the kernel hand a large buffer to the NIC, but the wire still has to contain valid TCP segments whose sequence numbers, checksums, options, and MSS boundaries make sense to the peer. Generic receive offload can merge adjacent incoming segments before the stack sees them, but must not merge across incompatible headers or invent stream order. RSS can spread flows across queues, but a single TCP flow normally needs stable ordering semantics. Hardware timestamps are useful only if the event corresponds to the packet boundary and direction the stack thinks it is measuring.
This is also why TCP bugs often look like impossible application bugs. A bad sequence-number update in a fast path can corrupt a stream long after the wrong packet left the CPU. A driver that mishandles a TSO descriptor can produce valid-looking frames with incorrect payload ranges. A receive path that marks a checksum as verified when it was not can let corrupted bytes enter a reliable stream. The application asked for bytes; everything below it is responsible for making the packet machinery disappear without changing the byte stream's meaning.
TCP's abstraction is therefore both simple and demanding. To the application it is "write bytes here, read bytes there." Underneath, it is a distributed protocol that continuously reconciles two endpoint views of sequence space, buffer space, and path capacity over an unreliable packet service. The rest of the chapter is the machinery behind that reconciliation.
Sources
7.2 Connections: handshake, state machine, teardown
TCP connection setup is not just a polite greeting before data transfer. It creates shared state in two kernels: each side learns the other side's 32-bit initial sequence number, confirms that packets can travel in both directions, negotiates options, and binds a four-tuple of local IP, local port, remote IP, and remote port to a specific incarnation of a byte stream.
The usual opening is the three-way handshake. The active opener, normally the client, sends SYN with sequence number x and enters SYN-SENT. The passive opener has already done bind(), listen(), and is in LISTEN; when it accepts the SYN, it allocates embryonic connection state, replies with SYN,ACK using its own sequence number y and acknowledgment x+1, and enters SYN-RECEIVED. The active opener replies with ACK acknowledging y+1; after that both endpoints can be ESTABLISHED.
The +1 is not an accident. TCP sequence numbers name positions in a byte stream, but SYN and FIN deliberately consume one sequence number even when they carry no payload. That lets TCP order connection-open and connection-close events with the data. A SYN is considered before the first data byte in its segment; a FIN is considered after the last data byte. Without this, an old duplicate FIN or a retransmitted opening segment could not be unambiguously distinguished from data at the edge of the stream. For implementers, this is why SND.NXT advances after queueing a SYN or FIN, and why packet traces often appear to acknowledge "one byte" that was never application data.
The TCP state machine is the protocol's protection against confusing intent, packets, and application lifetime. CLOSED means no connection state exists. LISTEN is a passive socket waiting for a SYN. SYN-SENT is an active open waiting for a matching SYN,ACK or simultaneous SYN. SYN-RECEIVED has sent a SYN,ACK and is waiting for the final ACK. ESTABLISHED is the full-duplex data state. Closing then splits because TCP is full duplex: each direction can be shut down independently.
An active close sends FIN and enters FIN-WAIT-1. Once that FIN is acknowledged, it becomes FIN-WAIT-2, meaning "my send side is closed, but I can still receive." When the peer's FIN arrives, the active closer acknowledges it and enters TIME-WAIT. A passive close receives FIN first, acknowledges it, and enters CLOSE-WAIT; the local application may still write until it calls close() or shutdown(SHUT_WR). Then the stack sends its own FIN and enters LAST-ACK until that FIN is acknowledged. CLOSING is the rarer crossed-FIN case: both peers sent FIN before receiving the other's FIN, so each waits for its own FIN to be acknowledged. These states matter in C programs because a socket stuck in CLOSE-WAIT is usually not a network problem; it is an application that received EOF and failed to close.
TCP also permits simultaneous open. If both peers actively open the same four-tuple at the same time, each sends SYN, each receives a SYN while in SYN-SENT, each replies with SYN,ACK, and the connection can become ESTABLISHED. It is uncommon in client/server code, but it is part of the state machine and it explains why robust TCP code cannot assume "the side that sent the first packet is the client" at every layer. Simultaneous close is more common conceptually: both applications decide to close, both FINs cross, and the connection passes through CLOSING before TIME-WAIT.
A half-close is not an error. shutdown(fd, SHUT_WR) sends a FIN for the local-to-remote byte stream while preserving the receive direction. Protocols that send a request and then wait for a large response can use this to say "there is no more request body" without discarding the response. At the API boundary, a received FIN becomes EOF: read() returns 0 after buffered data is drained. It is not the same as a reset.
RST is TCP's abort mechanism. A reset says that the referenced connection state is invalid or that the endpoint refuses to continue. Connecting to a closed port usually produces RST; writing to a connection after the peer has aborted may surface as ECONNRESET or EPIPE depending on timing and API. Unlike FIN, RST does not politely close one direction after all prior bytes are delivered. It tears down state and tells the application the stream was aborted. Low-level code must treat this distinction carefully: a FIN is orderly EOF; a RST means some in-flight data may have been discarded.
TIME-WAIT is paid by the endpoint that performs the final active close, because it sends the last ACK. It waits for 2*MSL, twice the maximum segment lifetime. The first reason is stale duplicate suppression: if a later connection reuses the same four-tuple, delayed packets from the old incarnation must not be accepted as new data. The second is reliable teardown: if the final ACK is lost, the peer in LAST-ACK may retransmit its FIN; the TIME-WAIT endpoint still has enough state to retransmit the final ACK instead of responding with a reset. The cost is visible on busy systems with many short connections: TIME-WAIT entries consume kernel memory and, for active outbound connects, can tie up ephemeral ports for that destination four-tuple until reuse is safe.
Server-side setup has its own practical queues. On Linux, the listen(fd, backlog) argument is not the number of half-open SYN-RECEIVED handshakes; since Linux 2.2 it controls the queue of fully established sockets waiting for accept(), capped by net.core.somaxconn whose documented default is 4096 in current kernels. Incomplete requests are governed separately by net.ipv4.tcp_max_syn_backlog; when SYN cookies are enabled there is no logical maximum for remembered incomplete sockets because the server can encode state into the SYN,ACK instead of keeping all of it in memory. This split explains a common production symptom: packet capture shows handshakes completing, but clients still stall because the accept queue is full and the application is not calling accept() fast enough.
For NIC and driver work, this lifecycle is not abstract. Hardware offloads can checksum, segment, steer by RSS hash, and timestamp packets, but the host TCP stack still owns these transitions and their timers. A driver bug that drops a final ACK, misreports checksum validity on a SYN,ACK, hashes related packets to the wrong receive queue, or delays interrupts under a burst can show up as SYN-RECEIVED buildup, spurious retransmits, CLOSE-WAIT leaks, or unexplained resets. Reading ss -tan state counts, packet timestamps, and per-queue drops is often the shortest path from "TCP is flaky" to the exact place where state stopped advancing.
Sources
7.3 Sequence numbers, ACKs, and SACK
TCP's sequence space is a byte coordinate system, not a packet counter. The Sequence Number field in the TCP header is 32 bits, and for a data segment it names the first byte carried by that segment. If a sender emits 1000 bytes beginning at sequence 50000, the segment covers the half-open range [50000, 51000). The next byte after the segment is 51000; that number is also the natural ACK value if the receiver now has a contiguous stream through byte 50999.
This byte numbering is what lets TCP survive arbitrary segmentation. The sender might hand the kernel a 64 KiB write, the kernel might keep it as one large skb until TSO, the NIC might put many MTU-sized frames on the wire, and the receiver might merge adjacent packets with GRO. None of that changes the logical stream coordinates. A retransmission of bytes [50000, 51000) uses the same sequence range as the original transmission.
The ACK field uses the same coordinate system. A cumulative ACK of N means: "I have received everything before N, and N is the next sequence number I still need." It does not mean "I received the segment whose sequence number is N." If the receiver has bytes [0, 1000) and then receives [2000, 3000), the cumulative ACK is still 1000, because byte 1000 is the start of a hole. TCP can buffer the out-of-order bytes, but it cannot advance the cumulative ACK past missing data without breaking the ordered-stream contract.
That single rule explains duplicate ACKs. Suppose the sender transmits three full-sized segments:
[1000, 2460)[2460, 3920)[3920, 5380)
If the middle segment is lost but the third arrives, the receiver cannot ACK 5380. It sends another ACK for 2460, often with SACK information if negotiated. From the sender's point of view, an ACK repeating the same cumulative value while new data is known to have left the network is evidence that later bytes arrived but an earlier range did not. Classic fast retransmit treats several duplicate ACKs, historically three duplicate ACKs, as a stronger loss signal than a single duplicate ACK, because ordinary reordering can also produce duplicates.
ACK generation is intentionally not one ACK per data segment. RFC 1122 says a TCP should implement delayed ACKs, with delay bounded below 0.5 seconds, and for a stream of full-sized segments should ACK at least every second segment. Modern stacks usually use shorter timers, but the principle matters: in a packet trace or low-latency benchmark, missing every-other ACKs are not automatically loss. Delayed ACK reduces packet rate and may piggyback on outbound data, but it also changes ACK clocking. Tiny request/response protocols that send one small segment then wait can accidentally pay a delayed-ACK timer unless TCP_NODELAY, TCP_QUICKACK, batching, or application framing is handled carefully.
Pure cumulative ACKs are compact but ambiguous when more than one range is missing. If bytes [1000, 2000) and [5000, 6000) are lost from the same flight, cumulative ACK alone tends to reveal one hole at a time. The sender may retransmit conservatively, wait for an RTO, or waste bandwidth retransmitting data already sitting in the receiver's out-of-order queue.
Selective Acknowledgment fixes that information problem without changing the meaning of the cumulative ACK. SACK is negotiated with the SACK-permitted TCP option in the SYN path. Once permitted, the receiver may attach a SACK option to established-connection ACKs. Each SACK block is a pair of 32-bit sequence numbers: the left edge is the first sequence number in a received out-of-order block, and the right edge is the sequence number immediately after that block. So an ACK might say cumulative 1000, with SACK blocks [2000, 5000) and [6000, 9000). Read that as: "I still need byte 1000, but I already have these later ranges."
The sender turns those reports into state. A real TCP sender keeps a retransmission queue of outstanding data: ranges sent but not cumulatively acknowledged. With SACK, it also maintains a scoreboard over that queue. Some ranges are known delivered, some are retransmitted and waiting for proof, and some are candidates for retransmission because enough higher sequence space has been SACKed or ACKed to imply a hole. Linux has evolved through SACK-based recovery, FACK-style accounting, and RACK/TLP time-based loss detection, but the core idea remains: reason in byte ranges and evidence.
This is where implementation bugs become expensive. If a driver corrupts skb length metadata, misplaces a GRO boundary, or reports checksum completion incorrectly, TCP may mark the wrong range as valid or invalid. If a capture point is before TSO or after GRO, one displayed "packet" may represent many wire segments or many coalesced receive segments. Sequence numbers are the stable way to reconcile those views.
Duplicate ACKs are also not proof of physical packet loss. Reordering can manufacture them. A later segment arriving before an earlier one forces the receiver to repeat the old cumulative ACK even if the earlier segment is merely delayed. GRO can hide this because several wire packets may be delivered upward as one aggregate, or not aggregated when a timestamp, sequence gap, checksum state, or header field prevents coalescing. RSS and receive steering are another source: RSS is meant to keep a flow on one receive queue so ordering is preserved while work is spread across CPUs. If hashing, encapsulation parsing, indirection-table changes, flow director behavior, or a virtualization layer sends one TCP flow to different queues, per-queue polling can reorder delivery and create duplicate ACKs.
The last trap is sequence wrap. TCP sequence numbers live in a finite 32-bit ring, so ordinary integer comparison is wrong near 2^32 - 1. A byte with sequence 10 may be after a byte with sequence 4294967290 in the same connection's current window. Kernel TCP code therefore uses modular comparisons, commonly named before() and after() in Linux-style code:
static inline bool before(uint32_t seq1, uint32_t seq2)
{
return (int32_t)(seq1 - seq2) < 0;
}
static inline bool after(uint32_t seq1, uint32_t seq2)
{
return before(seq2, seq1);
}
This idiom depends on comparing sequence numbers that are not more than half the sequence space apart, which TCP's window rules maintain. In low-level C, it is the difference between a retransmission queue that works for days and one that fails only when the counter wraps under production traffic. Use unsigned storage for the wire value, wrap-aware predicates for ordering, and half-open byte ranges for queue and scoreboard logic.
Sources
7.4 Loss detection and retransmission
Loss is not a packet saying "I am lost." It is an inference made at the sender from silence, repetition, or later evidence. TCP's sender has bytes outstanding between SND.UNA, the oldest unacknowledged sequence number, and SND.NXT, the next sequence number to send. If progress stops, or if ACKs imply that later data reached the receiver while an earlier hole did not, the sender retransmits. That inference is conservative because retransmission is also a congestion signal.
The oldest detector is the retransmission timeout, RTO. When TCP sends data and has no data waiting for acknowledgment, it arms a timer. If the timer expires before the data is cumulatively acknowledged, TCP retransmits the oldest unacknowledged segment. RFC 6298 defines the estimator. The sender measures a round-trip time sample R, maintains smoothed RTT SRTT, maintains deviation RTTVAR, and computes:
RTO = SRTT + max(G, 4 * RTTVAR);
Here G is the clock granularity. The first RTT sample initializes SRTT = R, RTTVAR = R / 2, and RTO = SRTT + max(G, 4 * RTTVAR). Later samples update RTTVAR with gain beta = 1/4 and SRTT with gain alpha = 1/8. Before a sample exists, RFC 6298 sets the initial RTO to 1s; if a computed RTO is below 1s, the RFC says it should be rounded up to 1s, though kernels may use lower minima after measurement. When an RTO fires, TCP backs off exponentially. On Linux, tcp_retries2 controls established-state retries and is documented with a default of 15, usually many minutes before the connection is declared dead.
Timeout recovery is blunt. It handles hard cases: a whole window vanished, the final segment of a small response was lost, ACKs stopped, or the sender has too little duplicate-ACK evidence. But an RTO is expensive. Classic congestion control treats it as severe congestion evidence: retransmit, collapse the congestion window, and restart cautiously. In a low-latency path, one timeout can dominate tail latency.
RTT measurement has a subtle ambiguity. Suppose segment X is sent, times out, and is retransmitted. When an ACK for X arrives, did it acknowledge the original transmission or the retransmission? If the sender guesses wrong, it can poison SRTT: a delayed original ACK can make the path look too fast, while a retransmission ACK can make it look too slow. Karn's rule avoids this by not taking RTT samples from retransmitted data.
TCP timestamps reduce that ambiguity. RFC 7323 defines the timestamp option, with TSval carried by the sender and TSecr echoed by the receiver. With timestamp-based RTT measurement, the echoed timestamp identifies which transmitted segment generated the ACK signal, so the sender can take samples Karn's rule would otherwise reject. This matters because software timestamps, hardware timestamps, pacing, TSO, interrupt moderation, and GRO can all shift where "send time" and "receive time" appear to be measured.
The second classical detector is fast retransmit. TCP ACKs are cumulative: if bytes 0..999 have arrived but byte 1000 is missing, the receiver keeps ACKing 1000 even if it receives later bytes. Those repeated ACKs are duplicate ACKs. One or two may just mean reordering. Three duplicate ACKs, in RFC 5681 Reno-style TCP, are enough evidence that a prior segment is probably missing. The sender retransmits without waiting for the RTO.
Fast retransmit is paired with fast recovery because duplicate ACKs also prove that data is still leaving the network and reaching the receiver. Instead of dropping immediately to one segment as timeout recovery does, Reno sets ssthresh to roughly half the flight size, retransmits the presumed missing segment, inflates cwnd for duplicate ACKs, and continues sending when allowed. When an ACK covers the recovery point, TCP deflates back to ssthresh and resumes congestion avoidance. SACK-based recovery refines this by using selective acknowledgment blocks to retransmit actual holes, but the core contrast remains: duplicate ACKs are a packet-clocked loss signal, while RTO is a silence signal.
Modern stacks add time-based loss detection before timeout. RACK-TLP, standardized in RFC 8985, treats a segment as lost when a later segment sent sufficiently after it has been acknowledged. RACK, Recent ACKnowledgment, uses transmit times and ACK/SACK feedback to infer that an older packet should have arrived by now. This works better than duplicate-ACK counting under reordering, small writes, or lost retransmissions. TLP, Tail Loss Probe, sends a timed probe near the end of a flight to elicit ACK feedback and convert what would have been an RTO into fast recovery. Linux documents tcp_early_retrans as enabling TLP by default with value 3, and tcp_recovery as enabling RACK with bit 0x1 by default.
Spurious retransmissions are the price of inference. A sender may retransmit data that was not actually lost because ACKs were delayed, reordered, aggregated, or observed late by the stack. Interrupt coalescing is a common low-level contributor: the NIC may receive ACKs promptly but delay the interrupt, so TCP samples a later receive time. GRO can merge receive processing into larger batches. Hardware timestamping can reveal wire timing, while software timestamping often measures a later driver or kernel point. On transmit, TSO lets the stack hand one large skb to the NIC for many wire segments; one software send time for that aggregate is only approximate. Those approximations feed SRTT, RACK reordering windows, pacing, and timeout behavior.
For NIC and driver work, this is not abstract protocol trivia. A lost transmit completion may make the kernel believe data is still queued or distort byte accounting. A receive checksum bug can cause the stack to discard a valid ACK or data segment; from TCP's point of view, that looks like network loss. A DMA mapping error, descriptor race, ring overrun, bad RSS steering decision, or incorrect offload metadata can create holes that TCP reports as retransmissions and congestion. Conversely, a packet capture taken at the wrong point may blame the network for a retransmission caused by local delayed ACK processing or timestamp distortion.
The practical debugging question is not only "was there a retransmission?" but "which detector fired, and what evidence did it see?" ss -i exposes rto, RTT, congestion state, retransmission counters, and delivery-rate estimates on Linux. Packet captures can distinguish timeout retransmissions from fast retransmits and TLP probes if taken close enough to the sender. Driver counters show drops, checksum errors, queue timeouts, and completion anomalies. Good TCP debugging correlates wire packets, kernel TCP state, and NIC ring events. When those disagree, the bug is often below the socket API.
Sources
7.5 Flow control and the sliding window
TCP flow control keeps a fast sender from overrunning a slower receiver. It does not protect routers, switches, or the path as a whole; that is congestion control, covered next. Flow control is end-host bookkeeping: how many more bytes can the receiver accept into its TCP receive queue?
The receiver answers that question in every TCP segment it sends back. The TCP header contains a Window field, historically a 16-bit unsigned value, that advertises available space beyond the next byte expected. If the receiver has consumed bytes up to RCV.NXT - 1 and advertises RCV.WND, then the sender may transmit sequence numbers in [RCV.NXT, RCV.NXT + RCV.WND), subject also to congestion control. On the sending side, outstanding data must not exceed the advertised receive window. In RFC notation, bytes below SND.UNA have been acknowledged, bytes from SND.UNA through SND.NXT - 1 are sent but not yet cumulatively acknowledged, and usable receive-window space is roughly SND.UNA + SND.WND - SND.NXT.
The word "window" is literal. The left edge is anchored by the oldest unacknowledged byte. When an ACK advances SND.UNA, the left edge slides right and frees room for more data. The right edge is SND.UNA + SND.WND; it moves when ACK progress and advertised-window updates change what the receiver says it can accept. A pure ACK can open the window when the receiving application finally calls read() and drains socket-buffer memory. Robust TCP code must treat sequence numbers, ACK numbers, and window edges as modulo-2^32 quantities, not ordinary signed integers.
static inline int before(uint32_t a, uint32_t b)
{
return (int32_t)(a - b) < 0;
}
That idiom matters in NIC and kernel work. A driver or offload path may be handling TSO super-packets, GRO completions, or hardware timestamped packets arriving out of cache-friendly order. The protocol still lives in byte sequence space. Misplacing a byte relative to the window edge becomes data corruption, stuck queues, or spurious retransmission.
The 16-bit header field is too small for modern bandwidth-delay products. Without extensions, the largest advertised receive window is 65,535 bytes, which cannot keep a single flow full on a high-rate or long-RTT path. RFC 7323 fixes this with the Window Scale option. The option carries a shift count from 0 through 14; after the handshake, the advertised header field is interpreted as SEG.WND << shift, expanding the receive window to a 30-bit quantity with a practical maximum around 1 GiB.
Window scaling has one sharp operational rule: it is negotiated only in the SYN exchange. A Window Scale option outside a SYN is ignored, and the window field in SYN and SYN-ACK segments is not scaled. Each direction has its own scale because each host advertises its own receive capacity. A program that wants a very large receive buffer should set SO_RCVBUF, or rely on defaults that permit a large one, before the connection opens. Once the scale is fixed, later autotuning can grow the receive buffer only within that negotiated scale.
Linux normally performs receive-buffer autotuning when tcp_moderate_rcvbuf is enabled. The kernel starts from configured defaults and adjusts the socket receive buffer up to limits such as tcp_rmem[2], memory pressure, and application settings. Autotuning changes how much memory the receiver will commit, and therefore how large an rwnd it can advertise, but the sender still sees only ACKs and scaled window updates. This is why throughput can improve when a benchmark raises net.ipv4.tcp_rmem, pins the reader on a less contended CPU, or drains the socket more aggressively.
A zero receive window is TCP's explicit back-pressure signal. It says: "I have no buffer space for new data at the current left edge." The sender must stop sending new data beyond the window, but it cannot simply sleep forever, because the packet that reopens the window might be lost. TCP therefore uses zero-window probes driven by the persist timer. RFC 9293 says the first probe should be sent after the zero window has existed for roughly the retransmission timeout, with later probes backing off exponentially. These probes are not congestion-loss retransmissions or TCP keepalives. Their job is to force the receiver to repeat the current ACK and window value.
Zero-window behavior separates protocol health from application health. If a capture shows a receiver repeatedly advertising win 0, the network may be fine. The application may be stalled, CPU-starved, blocked on disk, slow to decrypt, or using a too-small socket buffer. On a busy server, the cause may be NAPI budget, copy-to-user overhead, or lock contention delaying socket drains.
Silly Window Syndrome is the inefficient corner case where the window advances in tiny increments and the sender fills each opening with tiny segments. It can be caused by a receiver that advertises every small amount of freed buffer space, or by a sender that emits small writes without coalescing. The result is poor payload-to-header ratio, high packet rate, and wasted CPU. TCP avoids this from both ends. Receiver-side SWS avoidance withholds small window updates until it can advertise useful space. Sender-side SWS avoidance waits until it can send at least an MSS-sized segment, all queued pushed data under the right conditions, or a substantial fraction of the observed maximum window; Nagle's algorithm complements this by coalescing small writes when unacknowledged data is outstanding.
This is also where TCP_NODELAY gets misdiagnosed. Disabling Nagle can be right for latency-sensitive request/response protocols that already frame and batch carefully. It is not a cure for a receiver that advertises tiny windows because the application reads one byte at a time. Flow control is telling you about receive-side capacity. Congestion control is telling you about path capacity. The sender's effective flight limit is the minimum of the two: min(rwnd, cwnd), plus details such as pacing, MSS, and send-buffer availability.
When diagnosing throughput, ask which limit is active. An rwnd-limited flow has room in the congestion window but cannot send because the receiver's advertised window is small or closed. You see small advertised windows, zero-window events, persist probes, or sender-side metrics such as rwnd_limited time. A cwnd-limited flow has receive-window space available but is constrained by loss recovery, congestion avoidance, pacing, or startup behavior. Packet captures show the advertised window staying open while flight size tracks the congestion window. This distinction prevents a common mistake: increasing congestion-control aggressiveness or NIC queue depth when the actual bottleneck is a slow reader and a shrinking receive window.
Sources
7.6 Congestion control: AIMD, CUBIC, BBR
TCP reliability says that bytes will eventually arrive in order or the connection will fail. Congestion control answers a different question: how fast may the sender inject bytes into the network without causing persistent queues, drops, and congestion collapse? The key sender-side variable is the congestion window, cwnd. Unlike the receiver window, rwnd, which protects receiver memory, cwnd protects the path. At any instant the sender's usable flight limit is effectively min(rwnd, cwnd): it may not have more unacknowledged data in flight than either the receiver has advertised or the congestion controller believes the network can carry.
This is why a 100 GbE NIC does not automatically make a single TCP stream run at 100 Gbit/s. If the path RTT is 40 ms, filling a 100 Gbit/s bottleneck requires about 500 MB of data in flight. The NIC can DMA descriptors quickly, and TSO can turn a large socket buffer into many wire packets, but the TCP stack must still respect cwnd. The limit may be a path-control variable in struct tcp_sock, not PCIe bandwidth or ring size.
A new or recently idle connection cannot know the safe rate. TCP therefore begins with slow start, whose name is historical: it is slow compared with blasting a whole receiver window, but its growth is exponential. RFC 5681 defines slow start as the phase used when cwnd < ssthresh. For each ACK that cumulatively acknowledges new data, the sender increases cwnd by at most one sender maximum segment size, SMSS; over a round trip, the window approximately doubles. Slow start ends when congestion is detected, the receiver window becomes limiting, or cwnd reaches ssthresh.
Past ssthresh, the sender enters congestion avoidance. Classic TCP no longer doubles the window; it probes cautiously. The Reno-style rule is additive increase: grow by about one SMSS per RTT. On loss, infer that the path was overfilled and apply multiplicative decrease, usually cutting the window estimate in half. That combination is AIMD. Competing AIMD flows that see the same bottleneck tend to converge toward a fair share, while a congestion signal rapidly removes load from the queue.
Reno and NewReno are the canonical loss-based algorithms. Reno introduced fast retransmit and fast recovery around duplicate ACKs, avoiding a full timeout for many single losses. NewReno improves recovery when multiple packets are lost from one window by using partial ACKs to continue retransmitting without requiring SACK. They expose TCP's control-loop shape: ACK clocking drives increases; loss or ECN drives reductions; timeouts are more severe because they indicate that the ACK clock may have collapsed.
The weakness of Reno becomes obvious on high-bandwidth, high-delay paths. Additive increase of one segment per RTT can take a long time to recover a large window after a reduction. A 10 Gbit/s transcontinental flow may need many megabytes in flight; after a loss, Reno's linear climb can leave the bottleneck underused for many RTTs. This is the classic high-BDP problem.
CUBIC, standardized in RFC 9438 and long used as the Linux default congestion controller, changes the growth function while keeping the broad loss-based contract. Instead of increasing linearly with each RTT like Reno, CUBIC computes a target window from a cubic function of elapsed time since the last congestion event. The curve grows quickly when far below the previous pre-loss window, flattens as it approaches that old operating point, and then probes above it. The goal is to regain a known-good rate efficiently, spend time near the previous knee of the path, then search for more capacity.
That time-based growth is a major reason CUBIC works better on fast, long-distance networks. It scales window growth for high-BDP paths while retaining Reno-friendly behavior where Reno is adequate. Linux's choice of CUBIC reflects a practical deployment tradeoff: servers need a general-purpose default that performs well over ordinary Internet paths and large-BDP links. For NIC and driver work, packet pacing, segmentation offload, completion batching, and interrupt coalescing are serving a controller whose bursts and window changes may not look like Reno's simple sawtooth.
Not all congestion control is purely loss-based. Reno and CUBIC primarily treat packet loss as the hard congestion signal, though they can also respond to ECN. Delay-based algorithms look for rising RTT as evidence that queues are building before drops. Model-based algorithms try to estimate the path directly. BBR, short for Bottleneck Bandwidth and Round-trip propagation time, estimates the bottleneck delivery rate and a minimum RTT, then uses that model to choose a sending rate and bound inflight data. Rather than asking "how much can I increase before loss?", BBR asks what bandwidth-delay product the path appears to have.
BBR's model-based design can perform well where loss is not a clean congestion signal, such as links with shallow buffers, policers, or non-congestion random loss. It can also reduce standing queues compared with loss-based controllers that fill buffers until drops occur. But it depends on measurement quality, pacing, and interactions with competing traffic. In Linux, BBR relies on delivery-rate sampling and benefits from pacing support; timestamping, transmit scheduling, GSO/TSO sizing, qdisc choice, and driver queue behavior can affect whether the intended pacing reaches the wire.
ECN adds a congestion signal that avoids treating drops as the only way for the network to speak. RFC 3168 defines two ECN bits in the IP header. A sender marks packets as ECN-capable with ECT(0) or ECT(1). A congested router may mark the packet CE, Congestion Experienced, instead of dropping it. The TCP receiver echoes that mark with ECE; the sender reduces its congestion window and sets CWR. ECN still requires endpoint and network support, but the principle is powerful: signal congestion while preserving the packet and ACK clock.
TCP's send rate is a control loop, not a property of the API call that wrote the bytes. Applications enqueue data; the stack decides when data may be sent; the NIC executes the schedule under offload and queueing constraints. Loss-based controllers such as Reno and CUBIC probe until drops or ECN marks tell them to back off. Delay-based and model-based controllers try to infer trouble earlier or estimate the path's operating point. When debugging throughput, retransmits, latency spikes, or uneven flow distribution, cwnd, ssthresh, RTT, pacing rate, ECN counters, and bytes in flight are often more revealing than raw link speed.
Sources
7.7 Throughput, the BDP, and bufferbloat
TCP throughput is not determined by link rate alone. A 100 Gbit/s NIC transmits at line rate only when the sender may keep enough bytes outstanding, the receiver can accept them, and the path can carry them without building a damaging queue. The first-order bound is simple: useful throughput is at most the amount of data in flight divided by the round-trip time. In TCP terms, the sender is limited by the smaller of the congestion window, cwnd, and the receiver's advertised window, rwnd; RFC 5681 calls the actually outstanding, unacknowledged data FlightSize. If min(cwnd, rwnd) is too small, ACK clocking cannot feed the NIC fast enough.
That is the point of the bandwidth-delay product: BDP = bandwidth * RTT. It is the volume of data that occupies the path when the pipe is full. A path with high bandwidth or high RTT needs a large in-flight window, even when there is no loss. The units matter. 10 Gbit/s * 80 ms is 10,000,000,000 bit/s * 0.080 s = 800,000,000 bits, or about 100 MB. A single TCP flow over that path needs roughly 100 MB in flight to keep the bottleneck busy. With only a 1 MB usable window, the upper bound is 1 MB / 0.080 s = 12.5 MB/s, about 100 Mbit/s; the 10 Gbit/s interface will be idle roughly 99% of the time for that flow.
This is why window scaling was not optional polish for modern TCP. The original TCP header has a 16-bit window field, so the largest unscaled advertised receive window is 65,535 bytes. On the same 80 ms path, that caps a flow at about 6.55 Mbit/s, before congestion control or application behavior. RFC 7323's window scale option makes large receive windows representable, but the stack still needs memory policy, autotuning, and a congestion controller willing to grow cwnd to the path BDP. For low-level work, this shows up as socket buffer sizing, page accounting, DMA mapping pressure, descriptor ring occupancy, and whether a benchmark is measuring the NIC or a host-side limit.
The BDP is not a recommendation to fill every buffer with one BDP of extra packets. It says how much data should be in flight, including data on the wire and data already queued at the bottleneck. A small amount of buffering absorbs normal burstiness: interrupt moderation, scheduler jitter, TSO packets, and short application bursts. Too little buffering at the bottleneck can cause drops even when average offered load is near capacity. But a huge FIFO queue lets senders overshoot for a long time before loss appears, so delay rises instead of packets being dropped or marked promptly.
That failure mode is bufferbloat. Suppose a home router uplink is 20 Mbit/s and has 4 MB of transmit buffering. If a bulk TCP upload fills that queue, the serialization delay of the queued bytes alone is about 4 MB * 8 / 20 Mbit/s = 1.6 s. Interactive packets behind the upload wait in the same queue, so DNS, SSH, gaming, and ACKs see second-scale latency even though the link is not "down." Loss-based congestion control also receives a distorted signal: the queue absorbs excess sending for many RTTs, so the sender sees health until latency is already terrible. When the queue finally drops, the reaction is late and often synchronized.
Active Queue Management attacks the standing queue rather than merely increasing the FIFO. CoDel, standardized in RFC 8289, uses packet sojourn time, not just queue length, as the signal; Linux's tc-codel defaults use a 5 ms target and 100 ms interval. FQ-CoDel, standardized in RFC 8290 and exposed by Linux as fq_codel, combines flow queuing with CoDel so one bulk flow is less able to trap unrelated sparse traffic behind it. Fair queuing also makes the common user-visible case better: an SSH packet from a sparse flow can be scheduled ahead of a large backlog from a bulk transfer, while CoDel drops or marks packets from flows that maintain excessive delay.
Pacing is the sender-side complement. ACK-clocked TCP naturally spreads traffic when ACKs return smoothly, but modern hosts can create large bursts locally. With TSO/GSO, the kernel may hand the NIC a large TCP super-packet, and the NIC segments it into MTU-sized frames. That is excellent for CPU efficiency: fewer SKBs, qdisc operations, DMA mappings, and doorbells. It is also a burst source. A 64 KB TSO packet represents dozens of Ethernet frames that may leave back-to-back at line rate. On a fast NIC feeding a slower bottleneck, repeated TSO bursts can build queues even when the long-term TCP rate is correct.
Linux therefore has several layers that matter together. The fq qdisc is designed for per-flow pacing and honoring pacing requirements set by TCP. Congestion controls such as BBR depend heavily on pacing because they send at an estimated bottleneck rate rather than using loss as the primary clock. Loss-based controllers also benefit, especially after idle periods or clumped application writes. At the device boundary, Byte Queue Limits keep the driver/NIC transmit queue shallow by accounting bytes handed to hardware and bytes completed. The driver reports enqueue and completion with APIs such as netdev_tx_sent_queue() and netdev_tx_completed_queue(), allowing the kernel to stop stuffing the hardware ring once enough bytes are pending.
This is where TCP performance becomes driver engineering rather than only protocol theory. A deep TX descriptor ring can hide latency from the stack: packets already sitting in the NIC are beyond qdisc control, so AQM and pacing cannot reorder, delay, drop, or mark them intelligently. A shallow queue risks starving the device if completions, interrupts, or NAPI polling lag. BQL tries to find the minimum useful device backlog dynamically, leaving most queueing in software where qdiscs can apply policy. For a Solarflare-style low-latency NIC, line rate is easy to advertise, but low tail latency at high throughput requires tight control over where bytes wait.
The practical model is to separate window, pipe, and queue. The window must be at least the BDP for one flow to fill the path. The pipe is the useful in-flight data moving through the network. The queue is excess waiting at a bottleneck or device. Good tuning raises the first enough to fill the second while keeping the third short and visible to the scheduler. Bad tuning either leaves the NIC idle with an undersized window, or fills hidden buffers so every flow pays latency for a bulk transfer's convenience.
Sources
7.8 Options: scaling, timestamps, TFO, MPTCP
TCP's fixed header is deliberately small: ports, sequence numbers, acknowledgments, flags, window, checksum, urgent pointer. Almost everything that made TCP scale to high-speed, long-delay, mobile, and multipath networks was added through options. Options sit after the fixed 20-byte header, and the Data Offset field says where payload begins. Because Data Offset is only 4 bits and counts 32-bit words, the largest TCP header is 60 bytes, leaving only 40 bytes for all options and padding.
Options also have timing rules. Some describe a per-segment fact and can appear later. Others negotiate connection capability and only make sense during the SYN exchange, while both endpoints are constructing their TCP control blocks. If a stack misses such an option in the SYN or SYN-ACK, it generally cannot infer the peer's later behavior safely. This is why a SYN often carries MSS, SACK-permitted, Window Scale, Timestamps, sometimes TFO, and, for Multipath TCP, MP_CAPABLE.
MSS is the simplest and easiest to misuse. The Maximum Segment Size option advertises the largest TCP payload, in octets, that the sender of the option is willing to receive. It excludes IP and TCP headers. On a 1500 byte Ethernet MTU, ordinary IPv4 commonly advertises 1460; ordinary IPv6 commonly advertises 1440. MSS is directional. RFC 9293 requires TCP implementations to support the option, recommends sending it in SYNs when the receive MSS differs from the defaults, and requires default assumptions if no MSS arrives. The effective send size can still be smaller because path MTU discovery, encapsulation, options, or device limits reduce what should be put on the wire.
For low-level work, MSS is where protocol state meets DMA reality. With TSO, the kernel can hand a large skb to the NIC and ask hardware to split it into wire-sized TCP segments. The device must use the connection's MSS and header template correctly; a wrong option length, checksum setup, or segmentation size creates packets that look nearly valid but fail under offload.
Window Scale exists because the original TCP window field is only 16 bits. Without scaling, the largest advertised receive window is 65,535 bytes, too small for high bandwidth-delay product paths. The option carries a shift count: the advertised 16-bit field is interpreted as window << shift, so the wire field remains compatible while the internal receive window can be much larger. The shift is negotiated only in SYN segments and is fixed for the lifetime of the connection. Each direction has its own scale value. After establishment, a packet analyzer must remember the negotiated shifts; the raw Window field alone is not the usable receive window.
If a firewall or load balancer strips the SYN option, the endpoint falls back to an unscaled window. The connection still establishes, but throughput can collapse on long-RTT paths. Nothing later says "I wanted scaling but lost it"; you see small advertised windows and poor pipe fill.
SACK-permitted is another SYN-time capability bit. The two-byte option says that the sender can receive and process Selective Acknowledgment information, and it must not be sent on non-SYN segments. If negotiated, later ACKs may include SACK blocks describing received byte ranges beyond a hole. That changes loss recovery: the sender can retransmit missing ranges instead of guessing from cumulative ACKs alone. SACK blocks consume option space on ACKs, so timestamps plus several SACK ranges can fill the 40-byte area quickly. Packet parsers should treat TCP options as a variable-length list, not as a fixed struct overlay.
Timestamps add two 32-bit fields: TSval, the sender's timestamp value, and TSecr, an echo of the peer's timestamp value. The option is 10 bytes, usually padded for alignment. Timestamps are negotiated in the SYN exchange; after both sides agree, RFC 7323 requires timestamp options on non-SYN segments. Their first job is RTT measurement. A sender can place a timestamp in transmitted data, receive it back in TSecr, and compute a round-trip sample without relying on one outstanding segment per window.
Their second job is PAWS, Protection Against Wrapped Sequences. TCP sequence numbers are 32-bit. At high data rates, a connection can wrap that space within the lifetime of delayed duplicates. PAWS treats timestamp values as monotonically non-decreasing within a connection and rejects old-looking segments whose TSval is behind recent state. It is not a replacement for sequence checks; it is an additional age test. In C, use unsigned modular comparisons and be explicit about wrap behavior.
TCP Fast Open changes the cost model of connection setup. Ordinary TCP waits for the three-way handshake before delivering application data. TFO lets a client include data in the SYN on a later connection to the same server, authenticated by a server-issued Fast Open cookie. A first connection can request a cookie with an empty-cookie option; the server returns one in the SYN-ACK. On a later connection, the client sends SYN + cookie + data; if the cookie validates, the server can acknowledge and deliver the data before the handshake has fully completed. If validation fails, the server acknowledges only the SYN and the client retransmits data after the handshake.
TFO matters because many implementations and middleboxes assumed that SYNs do not carry application data. Accepting early data creates replay and resource-exhaustion concerns, so applications must tolerate possible replay and servers must account for work before full establishment. NIC receive paths, SYN filters, and accelerated listen queues also need to preserve or deliberately reject SYN payloads; silently dropping data while accepting the connection creates hard-to-debug latency cliffs.
Multipath TCP uses TCP options to make several ordinary-looking TCP subflows present one reliable byte stream to the application. The initial subflow negotiates capability with MP_CAPABLE; later subflows join with MP_JOIN; data sequence mapping options relate subflow sequence space to connection-level data sequence space. The design goal is compatibility: if the peer or path does not support MPTCP, the connection falls back to regular TCP. RFC 8684 treats absence or stripping of MPTCP SYN options as a reason to operate as single-path TCP.
That fallback behavior is a recurring theme. Some middleboxes strip unknown options, normalize SYNs, drop SYNs with data, or rewrite MSS. Good TCP extensions fail soft: no option means no feature, not a broken connection. For systems engineers, the implication is practical: capture the handshake. Small windows, no SACK recovery, missing timestamps, disabled TFO, or absent MPTCP subflows are often decided in the first two packets. At the NIC and driver boundary, option bytes are small, stateful, performance-critical protocol inputs.
Sources
7.9 TCP and the NIC: TSO, GRO, RSS, timestamps
TCP is specified as an end-to-end protocol, but on a modern host a surprising amount of TCP work is done at the boundary between the kernel and the NIC. That is acceptable only because the device and driver preserve TCP's contract: an ordered byte stream, correct sequence numbers and checksums on the wire, and feedback signals that do not invent loss, reordering, or timing behavior. NIC offloads are performance shortcuts around per-packet CPU cost; they are not permission to change TCP semantics.
Checksum offload is the simplest case. TCP's checksum is a 16-bit one's-complement checksum over the TCP header, payload, and an IP pseudo-header. The pseudo-header binds the segment to source and destination addresses and protocol, catching some misdelivery errors that a TCP-header-only checksum would miss. With transmit checksum offload, the kernel does not compute the final value before DMA. It fills the packet template and descriptor metadata that tell the NIC where the checksum field starts and what bytes are covered; the NIC writes the final checksum as it sends. On receive, the NIC can validate the checksum and mark the resulting skb so the stack can skip the full calculation.
This explains a common trace surprise: a capture inside the sending host may show bad outbound checksums because the capture point is before hardware completion. A capture on the wire should not. For driver work, the dangerous bugs are wrong offsets, IPv4 versus IPv6 pseudo-header mistakes, and stale checksum metadata after cloning, tunneling, or linearization.
TSO and GSO move the segmentation point. TCP's algorithms reason in bytes and MSS-sized wire segments, but the CPU does not want to traverse the whole stack for every 1448 or 1460 bytes of payload on a fast link. With TCP Segmentation Offload, the stack may hand the NIC a large skb, often tens of kilobytes, plus a TCP/IP header template and a gso_size equal to the MSS. The NIC emits ordinary TCP segments: each gets the right payload slice, IP length, TCP sequence number, flags as appropriate, and checksum. Generic Segmentation Offload is the software fallback that performs this late in the kernel when hardware cannot.
The key word is ordinary. TSO does not put a giant TCP segment on Ethernet. The wire still sees MSS-sized packets, and the peer's ACKs still describe byte ranges in the normal TCP sequence space. If a large skb covering bytes S..S+65535 is segmented at 1460 bytes, each emitted segment must advance seq by exactly the previous payload length. A descriptor bug here creates real malformed TCP: overlaps, gaps, or checksums over the wrong data.
TSO also changes observability. A capture above the device may show one large TCP packet because that is the pre-segmentation skb; an external tap shows the actual wire packets. When traces and counters disagree, ethtool -k and ethtool -K are often part of the first sanity check.
GRO and LRO are the receive-side mirror image. Generic Receive Offload coalesces adjacent incoming TCP segments from the same flow into a larger skb before the upper stack pays per-packet costs. If sequence numbers are contiguous and the headers are compatible, GRO can merge the data and pass one larger buffer upward. TCP still receives the same bytes in the same order; the coalesced skb is an internal batching unit, not a new wire format. Captures taken after GRO can therefore hide wire packet boundaries.
Large Receive Offload is a more aggressive hardware form of receive coalescing. It can be unsafe for routers, bridges, packet capture systems, and virtual switches because those paths may need original packet headers and boundaries. GRO is more conservative and stack-aware; it can flush when sequence continuity, ECN state, checksums, or header details make merging unsafe.
RSS is parallelism without accidental reordering. A high-end NIC exposes multiple receive queues, each with its own descriptor ring and interrupt path. Receive Side Scaling hashes packet fields, commonly the 5-tuple, through an indirection table to choose a queue. The goal is not just spreading load over cores. It is keeping one TCP flow on one queue so packets are processed in arrival order by a consistent CPU path.
If one flow is sprayed across queues, the host can manufacture reordering even when the network delivered packets in order. That can trigger duplicate ACK patterns that look like loss and distort ACK spacing measurements. For low-level C work, RSS means caring about hash keys, indirection tables, MSI-X vectors, queue affinity, and whether tunneled traffic is hashed on outer or inner headers.
Hardware timestamping moves time measurement closer to the event. Software receive timestamps are taken after interrupt moderation, driver work, and scheduling delay have already added jitter. Hardware timestamping records time at or near the MAC as a frame is transmitted or received, then reports it through mechanisms such as Linux SO_TIMESTAMPING. That is essential for PTP, latency measurement, and systems where a few microseconds of host jitter can dominate the result.
Timestamping metadata must remain associated with the right frame as descriptors complete, skbs are cloned, and packets are segmented or coalesced. With TSO, one large send request may become many wire frames, so "the transmit timestamp" must be interpreted according to what the NIC and driver actually timestamp: often a selected descriptor or packet, not every segment in the burst unless the hardware explicitly supports that.
Interrupt coalescing shapes TCP's feedback loop. A NIC can wait for several frames, or for a timer such as rx-usecs, before interrupting the CPU. This reduces interrupt rate and lets the driver process batches, improving throughput and CPU efficiency. It also delays packet delivery to TCP. For bulk transfer, batching can smooth CPU load and keep the pipe full. For latency-sensitive flows, it adds RTT jitter: ACKs may be generated or delivered in clumps, and the sender's ACK clock becomes partly a product of moderation policy.
That is why the same NIC may be tuned differently for storage replication, market data, RPC, and backup traffic. Disabling moderation may reduce tail latency but burn CPU. Increasing it may raise throughput while making RTT samples and pacing feedback less precise. The invariant is what matters: offloads may batch work, delay notifications, and change local buffer shapes, but they must preserve the TCP byte stream, ordering, valid wire headers, and feedback signals that loss recovery and congestion control depend on.
Sources
7.10 Observing and debugging TCP
TCP debugging starts by refusing to treat the connection as a black box. At any instant a sender is constrained by three gates: what the receiver advertised (rwnd), what congestion control permits (cwnd), and what the application put into the socket. Confused investigations mix these up. A fast NIC and a large negotiated window do not help if the application is asleep, cwnd has collapsed after loss, or the peer advertises a tiny receive window.
On Linux, start with ss -tiepm, usually filtered to one socket: ss -tiepm dst 192.0.2.10 dport = :443. The queue columns are the first clue. For an established TCP socket, Recv-Q is data received by the kernel but not yet read by the application. If it grows, the peer may soon become rwnd-limited because this socket cannot drain its receive buffer. Send-Q is data accepted from the local application but not yet acknowledged by the peer. A persistent Send-Q with rising retransmissions points toward network loss or congestion. A persistent Send-Q with a small advertised peer window points toward receiver-side backpressure. A near-zero Send-Q on a connection that "should be faster" often means the local application is not feeding TCP fast enough.
The -i details expose the control loop. rtt:<avg>/<rttvar> is the smoothed round-trip time and its variation, in milliseconds. High rtt is not by itself a fault; high or unstable rttvar makes the retransmission timer conservative and can turn bursts into stalls. cwnd is the sender's congestion window, normally printed in segments, so translate it through the current MSS before comparing it with a bandwidth-delay product. ssthresh is the boundary between slow start and congestion avoidance; a low value after a drop says loss recovery recently cut the connection down. retrans and timer fields tell you whether the kernel is waiting or repairing loss. bytes_acked separates a live but slow connection from a dead one: if it keeps advancing, forward progress exists.
Read these fields together. If cwnd is small, ssthresh is low, rttvar is high, and retrans increments, the sender is probably cwnd-limited by loss or congestion. If Recv-Q is high on the receiver and the sender's peer window is small or zero, the flow is rwnd-limited; the network may be innocent. If Send-Q is usually empty, bytes_acked advances only after writes, and ss reports delivery samples as application-limited on newer kernels, the bottleneck is above TCP. This matters in low-level work because the fix lands in different code: driver queueing and interrupt moderation for loss, socket-buffer sizing or reader scheduling for receiver pressure, batching and wakeup paths for application starvation.
Packet capture answers a different question: what did this host observe at a protocol boundary? Use tcpdump -i eth0 -nn -s 0 -w trace.pcap 'tcp and host 192.0.2.10', then inspect in Wireshark or with tcpdump -tttt -nn -r trace.pcap. Look for sequence holes, duplicate ACK trains, SACK blocks, zero-window probes, SYN retries, FIN/RST ordering, and whether resets come from an endpoint or middlebox. Capture both ends. A retransmission seen at the sender but not at the receiver is a path or capture-point problem; a retransmission seen at both ends with duplicate ACKs before it is usually real loss.
There is one major trap: host captures are not always wire captures. With TSO or GSO, the kernel and driver may hand the NIC a large skb and let hardware or late software segmentation produce the real MTU-sized TCP segments. With GRO, receive processing may merge incoming segments before capture. The result is a pcap containing 32 KB or 64 KB "TCP packets" that never existed on the wire, or missing the original packet boundaries that drove ACK clocking. For protocol truth, capture on a tap, switch mirror, or with offloads temporarily disabled using ethtool -K eth0 tso off gso off gro off, knowing this changes CPU cost and sometimes behavior. For NIC and driver engineers, the mismatch is central: bugs can live in descriptor setup, checksum metadata, gso_size, RSS steering, or GRO coalescing, while the pcap looks like a normal TCP problem.
Common pathologies have recognizable shapes. A retransmit storm is repeated repair of the same flight, often with cwnd collapse, rising rtt, duplicate ACKs or SACK blocks, and low useful bytes_acked. Causes include physical loss, overloaded queues, bad interrupt affinity, driver drops, or microbursts exceeding switch buffers. RST resets are abrupt aborts: TCP allows resets for nonexistent connections, unacceptable handshake ACKs, and application aborts. In traces, ask who sent the RST, whether its sequence number is acceptable, and whether it follows refused data, a closed port, a timeout, or a middlebox policy.
High connection churn produces a different failure mode. TIME-WAIT is not waste; it protects later incarnations of the same four-tuple from old duplicate segments. But a client opening many short connections can run out of ephemeral ports, and a server can accumulate many time-wait sockets. Watch ss -tan state time-wait, local port ranges, and whether connection reuse is possible at the application protocol. Do not treat tcp_tw_reuse as a generic performance knob; Linux documents it as reuse under protocol constraints, not a cure for poor connection management.
SYN-queue overflow appears before the connection is established. Symptoms include SYN retransmissions from clients, SYN-RECV buildup in ss -tan state syn-recv, kernel log messages about SYN flooding or syncookies, and nstat listen-overflow counters. The accept queue and SYN queue are related but not identical: a slow accept() loop fills completed children, while a SYN flood or handshake burst stresses half-open requests. somaxconn, the application's listen(backlog), tcp_max_syn_backlog, syncookies, and accept-loop CPU placement all matter.
Incast collapse is the data-center version of "everyone replied at once." Many senders transmit to one receiver after a barrier; switch and host queues overflow; losses synchronize; many flows cut cwnd; latency spikes; throughput falls even when average offered load looks reasonable. The fix is rarely "increase every buffer." You need pacing, request fan-in control, ECN where available, better queue management, or application-level staggering. At NIC level, inspect RX ring drops, interrupt moderation, NAPI budget pressure, RSS distribution, and whether one queue or CPU is the serialization point.
The discipline is to correlate layers. ss tells you what TCP believes. tcpdump tells you what a capture point saw. NIC counters tell you what hardware and the driver dropped or coalesced. Application logs tell you whether bytes were actually written and read. TCP is observable enough to debug well, but only if each observation is tied to the window, queue, timer, or offload boundary that produced it.
Sources