DeepConcepts

Networking / transport / tcp / congestion control

BBR's two estimates are never measured together, and the one that empties the queue is the one it lets go stale

The misconception

That BBR measures the bottleneck bandwidth and the round-trip propagation time and paces at exactly their product, so it fills the pipe without filling the queue, and that switching to it is a free upgrade over CUBIC. The two quantities cannot be sampled in the same state: a delivery-rate sample that shows the true bottleneck rate requires enough in flight to keep the bottleneck busy, and an RTT sample that shows the true propagation delay requires the queue to be empty. BBR resolves that by taking a windowed max of one and a windowed min of the other, which biases both toward keeping in-flight data high. If the queue never empties inside net.ipv4's 10-second min-RTT window, BBR's min_rtt is the propagation delay plus a standing queue, its bandwidth-delay product estimate is correspondingly too large, and it holds that queue open indefinitely. PROBE_RTT is the correction and it is not free: mainline Linux ships BBRv1, where bbr_cwnd_min_target is 4 packets and bbr_probe_rtt_mode_ms is 200, so a long-lived flow periodically drops to four packets in flight for at least 200 milliseconds. BBRv3, which is what most of the writing about BBR now describes, replaced that with a ProbeRTT cwnd gain of 0.5 — half the estimated bandwidth-delay product rather than four packets — and it is not in mainline.

18 min

BBR sets pacing_rate = pacing_gain * windowed_max(delivery rate, 10 round trips) and cwnd = cwnd_gain * that bandwidth * windowed_min(RTT, 10 seconds); both filters are one-sided, so the bandwidth estimate can only be raised by a sample taken while a queue exists and the round-trip estimate can only be lowered by a sample taken while one does not — which is why PROBE_RTT exists, and why on mainline Linux, which ships BBRv1, it cuts in-flight data to four packets for 200 milliseconds.

Where this is already explained

  • Your bandwidth-delay product used the wrong bandwidth and the wrong delay

    That the bandwidth-delay product is arithmetic you do once — the speed your link is sold at, times the number your ping prints — and that setting the socket buffer to the answer fills the pipe. Both inputs are usually wrong. The bandwidth is the narrowest hop on the path, not the link you pay for, and sizing to your own link rate on a path with a slower hop is worse than leaving the default alone. The delay has to be the unloaded minimum RTT, because throughput measured during a transfer multiplied by the RTT measured during the same transfer always returns the window you already had — so the rule certifies whatever queue you are already carrying instead of correcting it.

  • A speed test cannot see bufferbloat, and a smaller buffer is not the fix

    That bufferbloat is a hardware defect in somebody else's router, that a speed test will show it, and that the cure is a smaller buffer or a faster link. The queue is built by your own bulk flows, so a speed test — which measures exactly the quantity bufferbloat does not damage — reports the link as healthy while a video call on the same link is unusable. Shrinking the buffer trades latency for throughput and there is no single size that is right at two different link rates. And selecting fq_codel changes nothing at all unless you also shape a few percent below line rate, because until you do, the queue forms in the modem downstream of every qdisc you configured.

  • TCP Congestion Control

    That throughput is set by the link's bandwidth, so a slow transfer means buying more of it. A single flow is bounded by window/RTT, and the window is capped first by the receive window, then by the loss rate a loss-based controller has to operate against — upgrading 1 Gbps to 10 Gbps moves none of those, and oversized buffers along the path make the window grow far past the bandwidth-delay product, inflating latency while throughput stays flat.

  • Slow start is not slow, and your keep-alive connection is not warm

    That slow start is a slow warm-up phase you can wait out, that it ends early in any real transfer, and that a long-lived keep-alive connection stays warm. Growth is exponential, so slow start is the fastest thing TCP does; but its cost is counted in round trips rather than in seconds, which means a 1 MB response costs the same number of round trips on a 10 Mbps link and a 10 Gbps one, and for anything under a few megabytes slow start is not a phase of the transfer, it is the whole transfer. Linux then leaves slow start well below the bandwidth-delay product because CUBIC's HyStart exits on a delay rise once cwnd reaches 16, and net.ipv4.tcp_slow_start_after_idle defaults to 1, which halves the window once per retransmission timeout of idleness down to the initial 10 segments — so a connection used once a second is cold on every request.

4 published lessons depend on this concept, which is what moves it up the writing queue. Nothing is hidden behind this page — it has not been written.

Why this concept is on the site

Topics are chosen from places engineers visibly get stuck, and the sources are kept with the lesson so the claim is checkable.