Your bandwidth-delay product used the wrong bandwidth and the wrong delay

A gigabit link, an 80 ms path, and a file that moves at 26 Mbps. The slowest hop on that path is idle 73% of the time. Nothing is being dropped. The sender is waiting, and what it is waiting for is not bandwidth.

TCP is a sliding window protocol. The sender may have at most one window of unacknowledged bytes outstanding at any moment, and a byte is only released from that window when its acknowledgement — its ACK — comes back, one round trip later. So the rate of a single flow is a quantity divided by a delay: window over round-trip time, or RTT. Fill the window and stop.

The window that fills the path exactly is the bandwidth-delay product, or BDP: the rate of the path multiplied by its round-trip time. It is how many bytes are in the air at once when the path is running flat out. Set the window below it and the sender runs out of window before the first ACK arrives, so the link idles. Set it above and the extra bytes are not in the air at all — they are sitting in a queue at the slowest hop, adding delay and no throughput whatsoever.

The panel below is a packet model of one flow crossing one path. The sender injects while its window has room, the narrowest hop serves packets at exactly its own rate and tail-drops what will not fit in its buffer, and an acknowledgement returns one propagation delay after service. Every number underneath is measured from that, not evaluated from a formula. Start by dragging send window from left to right and watching the hero number rise, stop, and then fall over.

Three things to do, in order. One: drag send window up from 256 KB. Goodput rises in a straight line — 26.2 Mbps, then 52.4, then 96.5 — and then stops dead at 96.5 Mbps because that is everything the narrow hop has. The window where it stops reads 942 KB, and the window / BDP readout will tell you that is exactly one BDP. Keep dragging: the number does not improve, the RTT readout starts climbing, and past about 2.1 MB the goodput falls. Two: put the window back to 942 KB and drag the link you pay for from 1 Gbps to 10 Gbps. Nothing moves at all; that slider only ever mattered as an argument to a minimum. Three: set the sizing rule to my link rate × my idle ping, which is the calculation nearly every tuning guide prints. It asks for a 9.54 MB window and delivers 21.8 Mbps — worse than the 26.2 you started with, with 78% of everything offered thrown away. The log explains each verdict as it happens.

Rate times delay is not a measurement

The reason the third experiment fails is worth more than the arithmetic. Set the sizing rule to measured throughput × RTT during the transfer and the log runs the calculation three times. It returns the same window every time, whatever window you started from. From a 1.49 MB window it answers 1.49 MB, while the socket carries 50 ms of standing queue. From 256 KB it answers 256 KB, while the hop is idle 73% of every round trip.

That is not a bug in the rule. It is Little's Law. For any queueing system in steady state, items in the system equals arrival rate times time in system — and for a TCP flow the items in the system are the window. So throughput measured during the transfer, multiplied by the RTT measured during the same transfer, reconstructs the window that produced them. It is an identity. It has no information in it about whether that window is the right one.

A number you can act on has to come from something the transfer is not currently doing to itself. That means both inputs must be properties of the path:

• The rate is the capacity of the narrowest hop, which is what the flow could have if the window were right — not what it is achieving now.
• The delay is the minimum RTT ever observed on the path, which is propagation plus serialisation with no queue in it — not the RTT this transfer is currently inflating.

This is exactly why ss reports minrtt alongside rtt, and why the gap between them is the single most useful number on a slow socket. It is also why BBR keeps a windowed minimum of RTT over ten seconds and a windowed maximum of delivery rate rather than averages of either: an average RTT is contaminated by the queue the flow built, and an average delivery rate is contaminated by the moments the flow was not trying.

Which bandwidth: the smallest number on the path, not the one on the invoice

A path is a chain of links and the flow can only go as fast as the slowest one, so the bandwidth in the product is a minimum over hops. The simulation puts that in front of you: with your link at 1 Gbps and the narrow hop at 100 Mbps, the BDP is 943 KB. Drag the link you pay for across its whole range, from 10 Mbps to 10 Gbps, and above 100 Mbps nothing about the result changes at all.

Sizing to your own link rate is not merely useless, it is actively harmful, and the second experiment shows how much. Your link is 1 Gbps and your idle ping is 80 ms, so the rule asks for 9.54 MB. That window is 10.4 times the BDP, and four and a half times what the pipe and the whole buffer can hold between them, so the hop discards 78% of everything offered, the sender spends most of its capacity re-sending bytes it already sent, and goodput lands at 21.8 Mbps — below the 26.2 Mbps you had before you tuned anything. Meanwhile the measured RTT has gone from 80 ms to 179 ms, so every other flow sharing that hop now waits an extra 99 ms behind you. This is the shape of bufferbloat, reached here by a spreadsheet rather than by a router vendor.

You will rarely be told the narrow hop's rate, so measure it. The cleanest field test is to compare one stream against many:

iperf3 -c host -t 30          # one stream: limited by one window / RTT
iperf3 -c host -t 30 -P 10    # ten streams: limited by the path

If ten streams get roughly ten times what one stream got, the path has capacity that a single window is not reaching, and the aggregate is your estimate of the narrow hop. If ten streams get the same total as one, you have found the actual capacity and no amount of window tuning will move it. That one comparison separates the two diagnoses that this whole lesson exists to keep apart, and it takes a minute.

Note what the parallel-stream trick is really doing: n flows have n windows, so the aggregate in flight is n times one window. It is a way of getting the BDP filled by brute force when you cannot change one socket's buffers. It is also why "just use more streams" quietly makes bufferbloat worse when the window was not the problem.

One BDP is not what a real stack provisions, and buffer bytes are not payload bytes

So you have the right rate and the right delay, you multiply, and you set SO_SNDBUF to the answer. You will still not get line rate, for three reasons that live in the kernel rather than in the arithmetic.

First, Linux deliberately provisions two BDPs, not one. Receive buffer autotuning — Dynamic Right-Sizing, controlled by net.ipv4.tcp_moderate_rcvbuf, on by default — runs once per round trip in tcp_rcvbuf_grow(), and the first thing that function computes, once past the guard that skips it when autotuning is off, is rcvwin = newval << 1, where newval is the number of bytes the application actually read in the last round trip. The comment above it says why: /* DRS is always one RTT late. */ The measurement describes the round trip that has finished, and the window it sets governs the one about to start, so the kernel doubles it to leave room for the flow to have sped up in between. Try the 2 × narrowest hop × minimum RTT rule in the panel: goodput is the same 96.5 Mbps you got at one BDP, and the RTT is 160 ms instead of 80. The doubled window is a ceiling, not an operating point — it exists so that the receiver is never the thing that limits you, and it works only because something else, the congestion window, stops the sender well below it. Remove that, as the panel does, and the second BDP turns straight into standing queue.

Second, the send buffer is accounted in kernel memory, not in payload. tcp_sndbuf_expand() sizes the send buffer as sndbuf_expand × nr_segs × per_mss, where per_mss is the maximum segment size — MSS, the largest payload TCP will put in one packet — plus MAX_TCP_HEADER plus the shared-info struct, rounded up to the next power of two, plus an aligned sizeof(struct sk_buff). On a typical x86-64 build MAX_TCP_HEADER is 256 and the shared-info struct is 320, so a 1448-byte segment reaches about 2,024 bytes before the rounding — just under the power of two — rounds up to 2,048, and then gains another 256 for the sk_buff. The socket is charged roughly 2.3 KB of buffer for 1.4 KB of data. Those two constants depend on build options, so treat the ratio as the point and not the exact byte count. The multiplier sndbuf_expand is 2 for CUBIC — the comment says CUBIC needs 1.7 and it is rounded up for cushion — and 3 for BBR, whose own bbr_sndbuf_expand() returns 3 "since BBR may slow-start even during recovery". A send buffer set to exactly one BDP of payload therefore holds well under one BDP of outstanding segments.

Third, the ceiling is much lower on the send side than the receive side, and it is the send side that people forget. tcp_init() sets tcp_wmem[2] = max(64 KB, min(4 MB, limit)) and tcp_rmem[2] = max(128 KB, min(32 MB, limit)), so on any machine with real memory autotuning may grow a receive buffer to 32 MB and a send buffer only to 4 MB. Four megabytes over a 180 ms London-to-Singapore path is 186 Mbps, and that is the hard ceiling for one socket before congestion control has said anything at all. Every tuning guide tells you to raise tcp_rmem. The one that is eight times tighter is net.ipv4.tcp_wmem, on the sending machine, which is frequently not the machine anyone logged into.

There is a fourth item that used to be on this list and is not any more, which matters because the advice outlived the mechanism. For twenty years the rule was "double your tcp_rmem, because tcp_adv_win_scale reserves half the buffer for packet overhead". The kernel documentation now opens that parameter's entry with one line — tcp_adv_win_scale - INTEGER: Obsolete since linux-6.6 — before describing what it used to do. Since 6.6 the split is measured per socket instead of assumed: the kernel keeps a scaling_ratio in 256ths, updated in tcp_measure_rcv_mss() from the actual skb->len / skb->truesize of arriving packets, and initialised to TCP_DEFAULT_SCALING_RATIO, which is 128 — the old 50% assumption, kept only as a starting guess. A receiver taking full-sized frames off a modern NIC settles well above 50% and gets more window per byte of buffer than the old rule promised; a receiver taking a stream of small segments settles far below it and gets much less.

Which is the honest summary of this whole section: the number of payload bytes you get to have outstanding is not a number you can compute from the outside. You set the ceilings and read back what the socket actually did.

Checking it on a real system

Run ss -ti against the live socket while the transfer is moving. Everything you need is in one line.

• minrtt: is your delay input. It is the lowest RTT the socket has ever seen, which is as close to an unloaded measurement as you will get without stopping the transfer. Multiply it by your estimate of the narrow hop's rate; that is the BDP, and it is the only version of the calculation worth doing.
• The gap between rtt: and minrtt: is the standing queue you are currently causing. minrtt:18 rtt:214/6 is 196 ms of self-inflicted delay and a window roughly twelve BDPs too large.
• cwnd: times mss: divided by minrtt: is the ceiling the congestion window is currently imposing. If that matches your measured throughput, buffers are not your problem and you should be reading about loss, not about window sizing.
• rcv_space: is what receive autotuning has settled on, and wscale: shows the negotiated shifts. A wscale:0,0 on a long path means window scaling never got negotiated and your window cannot exceed 65,535 bytes no matter what any buffer is set to.
• delivery_rate is the kernel's own estimate of the narrow hop, computed the way BBR computes it. It is usually a better input to the product than anything you will get from a speed test, because it is measured on the path this socket is actually using.

To reproduce any point on the sweep chart on real hardware, impose the path with netem and then walk the buffers:

tc qdisc add dev eth0 root netem delay 40ms      # 80 ms RTT
sysctl -w net.ipv4.tcp_wmem='4096 16384 8388608'  # raise the 4 MB ceiling
ss -ti dst 10.0.0.5 | grep -o 'minrtt:[0-9.]*'  # your delay input

Two rules for the measurement itself. Never take the RTT input from a ping run during the transfer — that is the identity again, and it will confirm whatever you already have. And never set SO_SNDBUF or SO_RCVBUF unless you intend to freeze the value: setting either one takes the socket out of autotuning permanently, so a well-meaning line of code that pins 256 KB has capped every long-haul transfer that process will ever make at 256 KB divided by the RTT. Raising the sysctl ceilings and letting the kernel tune inside them is almost always the better move.

Next: the option that has to be negotiated in the SYN before any of this is reachable, window scaling; what the window does before it reaches the BDP at all, in slow start; who chose the buffer that decides how far above the BDP you sit, in bufferbloat; and the controller that measures the two inputs to this product continuously instead of being told them once, BBR.