Linux does not implement RFC 896, and TCP_NODELAY is not what fixed your latency

The 40 millisecond stall people blame on Nagle's algorithm is real, and TCP_NODELAY does remove it. Both of the things you probably believe about why are wrong: Linux does not implement the rule RFC 896 specifies, and the 40 ms is not a constant the network charges you. It is a property of the shape of your writes, and you can move it by one byte.

Two mechanisms have to line up for the stall to happen. On the sending side, Nagle's algorithm refuses to put a segment smaller than one maximum segment size (MSS) on the wire while an earlier segment is unacknowledged, so that an application writing four bytes at a time does not produce a packet per keystroke. On the receiving side, the delayed acknowledgement holds the ACK back for up to 40 ms hoping to carry it on a reply, so that a request-response protocol sends two packets per exchange instead of three. Each is a reasonable optimisation. Put them on the same connection and the sender is waiting for an acknowledgement the receiver is deliberately not sending.

The panel below is not a diagram of that. It is both stacks: a sender with snd_una, snd_nxt and snd_sml running Linux's tcp_nagle_check on every partial segment, and a receiver with a quick-acknowledgement budget, a ping-pong counter and an adaptive timeout running __tcp_ack_snd_check on every arrival. The latency falls out of those two state machines talking to each other. Start by dragging body size one stop, from 1,447 bytes to 1,448.

Three things to do, in order. One: leave everything as it loads and move body size from 1,000 bytes to 1,447. Nothing happens — the median is 40.5 ms either way. Now move it one more stop, to 1,448, and the median drops to 0.30 ms. One byte of payload removed a 40 ms latency floor, and no socket option was involved. Two: put the body back to 1,000 bytes and switch write pattern to writev(header, body). The median is 0.30 ms again, and the client sent 24 packets instead of 48. Three: put the pattern back to write; write; read and tick TCP_NODELAY. Same 0.30 ms — and 48 packets. Both fixes remove the stall. Only one of them also stops sending twice as many packets as the request needs, which is the thing Nagle was put there to prevent.

The rule Linux runs is not the rule you have read

RFC 896, John Nagle's 1984 note, states the algorithm in one sentence: The solution is to inhibit the sending of new TCP segments when new outgoing data arrives from the user if any previously transmitted data on the connection remains unacknowledged. RFC 1122 §4.2.3.4's discussion restates it the same way, with the one exemption that keeps bulk transfer working: If there is unacknowledged data (i.e., SND.NXT > SND.UNA), then the sending TCP buffers all user data (regardless of the PSH bit), until the outstanding data has been acknowledged or until the TCP can send a full-sized segment (Eff.snd.MSS bytes; see Section 4.2.2.6). The same section makes the escape hatch mandatory: A TCP SHOULD implement the Nagle Algorithm [TCP:9] to coalesce short segments. However, there MUST be a way for an application to disable the Nagle algorithm on an individual connection. That is where TCP_NODELAY comes from — it is not a Linux extension, it is a requirement of the host requirements document. The panel's RFC 896 / RFC 1122 setting implements exactly that paragraph, full-sized-segment exemption included.

Any previously transmitted data. Linux does not test that. Here is the whole of tcp_nagle_check from net/ipv4/tcp_output.c:

• return partial &&
•   ((nonagle & TCP_NAGLE_CORK) ||
•    (!nonagle && tp->packets_out && tcp_minshall_check(tp)));

and tcp_minshall_check, which the comment above it labels Minshall's variant of the Nagle send check:

• return after(tp->snd_sml, tp->snd_una) &&
•   !after(tp->snd_sml, tp->snd_nxt);

snd_sml is the sequence number at the end of the last small segment the sender transmitted — tcp_minshall_update advances it only when skb->len < pcount * mss. So Linux blocks a partial segment when the last sub-MSS segment is unacknowledged, not when any segment is. A sender that has 400 kilobytes of full-sized segments in flight and then writes eight bytes puts those eight bytes on the wire immediately.

Reproduce it. Untick TCP_NODELAY, which the free preview left on, then move header size to 1,448 B — one exact full-sized segment — and leave the body at 1,000. The median is 0.30 ms and nothing stalls. Now switch the Nagle rule radio to RFC 896 / RFC 1122 without touching anything else: the median jumps to 40.5 ms and 23 of 24 exchanges hit the delayed-ACK timer. The write pattern did not change. The application did not change. The only thing that changed is which of two published descriptions of Nagle's algorithm the kernel is running, and it is worth 135 times the latency.

There is a second exemption, in tcp_write_xmit, which passes tcp_skb_is_last(sk, skb) ? nonagle : TCP_NAGLE_PUSH. Nagle is consulted only for the buffer at the tail of the write queue; anything behind another buffer is pushed unconditionally, because the comment says such frames have no chances to get new data. Both exemptions push in the same direction: Linux's Nagle blocks far less often than the specification says, which is why the stall feels arbitrary when you meet it. It is not arbitrary. It requires a partial segment at the tail of the write queue with an unacknowledged small segment ahead of it, and that is a much narrower target than any unacknowledged data.

Put the header back to 4 B and the rule back to Linux (Minshall) before going on; the numbers below assume both.

The receiver decides, and it decides in three tests

Nothing above forces a 40 ms wait. Nagle only holds data until an acknowledgement arrives, so how long the hold lasts is entirely the receiver's choice. Linux makes that choice in __tcp_ack_snd_check, which tests three things in order and acknowledges immediately if any of them is true.

• More than one full frame has arrived. (tp->rcv_nxt - tp->rcv_wup) > icsk->icsk_ack.rcv_mss, and — the conjunct that is easy to miss — the right edge of the window has to advance far enough with it, or tcp_recvmsg() is left to send the acknowledgement instead. This is RFC 1122 §4.2.3.2's requirement that in a stream of full-sized segments there SHOULD be an ACK for at least every second segment, and it is why bulk transfer never stalls. Note what it is measured against: rcv_mss is not the negotiated MSS, it is the largest segment this receiver has actually seen. tcp_measure_rcv_mss starts it at TCP_MIN_MSS, which is 88 bytes, and raises it toward the advertised MSS as bigger segments arrive. The decision log prints where it ended up.
• The socket is in quick-acknowledgement mode. On the first data packet of a connection, tcp_event_data_recv calls tcp_incr_quickack, which grants a budget of rcv_wnd / (2 * rcv_mss) capped at TCP_MAX_QUICKACKS, 16. Each acknowledgement that covers new data spends one. While the budget lasts, every acknowledgement is immediate.
• A protocol state demands one — ICSK_ACK_NOW. Not reachable in this panel.

If none of the three fires, tcp_send_delayed_ack arms a timer and the sender waits. The catch is in the second test, because tcp_in_quickack_mode is icsk->icsk_ack.dst_quick_ack || (icsk->icsk_ack.quick && !inet_csk_in_pingpong_mode(sk)). The first disjunct is a route attribute, set with ip route ... quickack 1, and it is the one per-path override that does not need the application's cooperation. Along the ordinary path, budget alone is not enough. The socket must also not have decided that this connection is interactive, and it decides that in four lines of tcp_event_data_sent: if this host sends data less than ato after the last packet it received, the ping-pong counter goes up. net.ipv4.tcp_pingpong_thresh is 1. One fast reply, and the budget is dead weight for the rest of the connection.

That is what the first exchange in the panel is showing you. The exchange 1 readout says 0.50 ms while the median says 40.5 ms, and the difference is entirely that the server had not yet replied to anything when the first request arrived.

Now make the socket refuse to enter ping-pong mode, and watch what replaces it. Set server handler time to 60 ms, which is longer than ato, so tcp_event_data_sent's now - lrcvtime < ato test is never true. Then set exchanges to 8: 0 of 8 hit the delayed-ACK timer, and the median is 60.4 ms, all of it the handler. Now set exchanges to 24. Suddenly 16 of 24 stall and the median is 100 ms. The first eight exchanges spent the 16-acknowledgement budget, two per exchange, and from the ninth onward there is nothing left. Set exchanges to 17 and you can find the exact edge: 9 of 17.

This is the single best reason the interaction survives code review. A benchmark that runs ten requests measures the quick-acknowledgement budget. A service that runs for a week measures the steady state. They are different numbers and the loop is identical.

Put server handler time back to 0.1 ms and exchanges back to 24 before the next section.

The 40 ms is a constant, not a measurement

The delay is icsk_ack.ato, which starts at TCP_ATO_MIN and adapts toward the interval between arrivals. tcp_send_delayed_ack then bounds it, and the bound is the part that matters:

• int rtt = max_t(int, usecs_to_jiffies(tp->srtt_us >> 3), TCP_DELACK_MIN);
• if (rtt < max_ato) max_ato = rtt;
• ato = min(ato, max_ato);

The round-trip time can shorten the delay, but only down to TCP_DELACK_MIN, and TCP_DELACK_MIN is HZ / 25: 40 ms on every mainstream configuration. There is no path short enough to make it smaller. A loopback connection with a 0.05 ms round trip waits the same 40 ms as one across a campus.

Walk round-trip time and read the median at each stop, then tick TCP_NODELAY and read it again. At 0.2 ms: 40.5 ms against 0.30 ms. At 5 ms: 50.1 ms against 5.1 ms. At 20 ms: 80.1 ms against 20.1 ms. At 80 ms: 200 ms against 80.1 ms. The penalty at each stop is 40 ms of timer plus exactly one extra round trip, because the held write cannot leave until the acknowledgement arrives — 40.2 ms, 45 ms, 60 ms and 120 ms. The timer half never moves. The ratio does, from 135 times the unblocked latency on a datacentre path down to 2.5 times across a continent, which is why this gets discovered by people running services in one availability zone and dismissed by people testing across the internet.

The ceiling is TCP_DELACK_MAX, HZ / 5, 200 ms, which is where ato saturates when arrivals are far apart. RFC 1122 permits considerably worse: the delay MUST be less than 0.5 seconds. Linux is well inside the specification at both ends. The 40 ms is not a bug and it is not tunable through /proc — it is a compile-time constant, and the only per-socket control over it is TCP_QUICKACK.

Put round-trip time back to 0.2 ms.

Four fixes, and what each one costs

All four work in the panel. They are not equivalent.

TCP_NODELAY on the client. Median 0.30 ms, 48 packets for 24 exchanges, 502 request bytes per packet. It works by making the sender stop caring, which means the two-write pattern now puts two packets on the wire for every request forever. On one connection that is invisible. On a proxy holding fifty thousand of them it is the packet rate Nagle was written to prevent, and RFC 896's original argument — that tinygrams congest the network — was not wrong, it was just written about a network where 41-byte packets were a meaningful fraction of the load.

writev(), or any buffered writer. Set write pattern to writev(header, body) with TCP_NODELAY off. Median 0.30 ms, 24 packets, 1,004 request bytes per packet. Identical latency, half the packets, and the socket option is not involved at all. Nagle never gets a vote because at the moment the single write is pushed there is nothing unacknowledged to block on — the decision log says so on the the shape row. This is the fix, and the reason it is the fix is that the two-write pattern was never something you wanted; it is an artefact of how the serialiser was written.

TCP_CORK. Tick it with the pattern back on write; write; read. Median 0.30 ms and 24 packets, the same as writev, because that is what it is: the kernel accumulating your writes until you clear the option. Note tcp_nagle_check's first branch — (nonagle & TCP_NAGLE_CORK) is tested before the TCP_NODELAY branch, so cork is stronger than nodelay and a socket with both set behaves as corked. The hazard is the one Nagle does not have: cork holds partial data with no acknowledgement to release it, so an application that forgets to uncork holds it indefinitely.

TCP_QUICKACK on the server. Untick TCP_CORK first — the last fix left it on, and it hides this one — then set the selector to set once, after accept(). The median is 40.5 ms, exactly the same as off. This is the important negative result on this page. __tcp_sock_set_quickack calls inet_csk_exit_pingpong_mode, which zeroes the counter; and then the server's very next reply within ato calls inet_csk_inc_pingpong_cnt and puts it straight back, because the threshold is 1. Now set the selector to re-armed after every recv(): median 0.50 ms, 0 of 24 stalls. It does work — if you call setsockopt after every single read, forever. And notice the cost: 0.50 ms rather than the 0.30 ms the other fixes give, because the acknowledgement is now a packet of its own instead of riding on the response, and the client still waits a round trip for it.

The ranking follows from that. Fix the write pattern; it is free and it reduces packets. If you cannot — the writes are in a library you do not own — set TCP_NODELAY and accept the packet count. Use TCP_CORK only where you genuinely are assembling a message from pieces and will reliably uncork, which is the header-then-sendfile case it was added for. Use TCP_QUICKACK only when you control the receiving loop tightly enough to re-arm it, and know that you are paying an extra packet per exchange to do the same job.

One more boundary. Turn TCP_NODELAY off and everything else back to the defaults, then set body size to 16 KB. The median is 0.50 ms with 0 of 24 delayed-ACK stalls, even though the write pattern is still two writes and Nagle is still on. The first arm of __tcp_ack_snd_check is doing the work: a 16 KB body is twelve segments, more than one full frame arrives immediately, and the receiver acknowledges without consulting any timer. Nagle plus delayed acknowledgement is a small-message problem, and it stops being a problem at exactly the point the receiver has two full frames to acknowledge.

Checking it on a real system

The signature is a latency distribution with a hard floor at a multiple of 40 ms and nothing underneath it. Confirm it in this order.

• ss -ti on the socket while it is slow. rcv_mss: tells you what the receiver's more than one full frame test is being measured against, which is the number the first arm of __tcp_ack_snd_check compares to; ato: is the current delayed-acknowledgement timeout in milliseconds, and if it reads ato:40 you are looking at TCP_ATO_MIN. unacked: above zero on an otherwise idle connection is the sender holding data.
• nstat -az TcpExtDelayedACKs TcpExtDelayedACKLocked TcpExtTCPAutoCorking, or the same counters in /proc/net/netstat. Divide DelayedACKs by the request count. If it is near one per request, every request is paying a timer. TCPAutoCorking counts a different thing worth knowing about: net.ipv4.tcp_autocorking defaults to 1, and tcp_should_autocork withholds a partial buffer when the retransmit queue is non-empty and there is already an skb of yours in the qdisc or NIC queue (sk_wmem_alloc > skb->truesize) — not merely a bare acknowledgement, which the comment above it says it deliberately ignores. That is a third small-packet mechanism, it has no timer, and it is on by default.
• strace -tt -e trace=write,writev,sendto,sendmsg,recvfrom on the client. You are looking for two send syscalls with no read between them, and then a read that returns roughly 40 ms after the second send. That is the whole diagnosis, and it takes ten seconds. If the two writes are there, you have found the bug and it is in your code, not in the kernel.
• tcpdump -ttt -i any 'tcp port 5432' and read the delta column. The stall shows as a gap of 00:00:00.040 between your first segment and the peer's bare ACK, with your second segment immediately after it. A bare ACK arriving 40 ms after data, carrying no payload, is the delayed-acknowledgement timer and nothing else looks like it.
• Then check the obvious thing last, because it is the thing people check first: getsockopt(fd, IPPROTO_TCP, TCP_NODELAY). Some runtimes set it for you and some do not, and the difference is not guessable. Go's newTCPConn calls setNoDelay(fd, true) on every connection it constructs, and net/tcpsock.go documents it: The default is true (no delay), meaning that data is sent as soon as possible after a Write. .NET's TcpClient.NoDelay, by contrast, is a plain pass-through to the socket option with no default of its own, so it inherits whatever the operating system does, which is Nagle. Check the value on a live socket rather than reasoning about your runtime. If it is already set, the 40 ms you are looking at is something else, and slow start or your own server's queueing is where to look next.

The confirming experiment is a one-line change: replace the two writes with one and re-measure. If the floor disappears, it was this. If it does not, nothing you do to socket options will help, and it is worth knowing that before you ship a configuration change.

Related: the other reason a small response sits still on an idle link is slow start, which withholds bytes the sender is allowed to send rather than bytes the application has already written; the opposite failure, where the network holds far too much of your data instead of too little, is bufferbloat; and the window arithmetic underneath both is the bandwidth-delay product. None of them is fixed by a socket option, which is the pattern worth taking away.