A speed test cannot see bufferbloat, and a smaller buffer is not the fix

A speed test cannot detect bufferbloat. It is not that speed tests are bad at it — they measure the one quantity bufferbloat leaves completely untouched. Your link really is delivering every megabit you pay for, and the video call on it really is unusable, and both of those are the same fact.

Bufferbloat is a standing queue of your own packets sitting in a buffer at the slowest hop of the path. A bulk transfer grows its congestion window — the amount of unacknowledged data it is allowed to have outstanding — until something on the path drops a packet. The path drops a packet when a buffer fills. So a loss-based sender does not stop at the point where the link is full; it stops at the point where the link is full and the buffer in front of it is also full. Every packet that arrives afterwards waits behind that entire buffer. If the buffer holds 512 KB and the uplink drains at 20 Mbps, the wait is 210 milliseconds, and it is paid by the video call, the DNS lookup and the SSH keystroke sharing the link — not by the transfer that built it.

The panel below is a two-queue model of one home connection on its upload direction. Your router has a queueing discipline — a qdisc, in Linux's spelling — that you can configure; behind it sits the modem, with a fixed unmanaged first-in-first-out buffer you cannot. One or more bulk uploads run flat out under CUBIC, the congestion control algorithm Linux has defaulted to since 2.6.19. A video call sends a 200-byte packet every 20 ms alongside them and measures how long each one takes to get out. Nothing here is a formula being plotted: packets are enqueued, served at a rate, dropped when a buffer is full, and the delay is whatever the queue in front of them happened to be.

Start with the controls exactly as they load, and read the two numbers at the top against each other. Then move shaper on your router one step at a time and watch which of them reacts.

Three things to do, in order. One: read the panel as it loads. The speed test says 19.60 Mbps out of 20 — 98% of the line, a result that would close any support ticket — while the call's median one-way delay is 187 ms and its 95th percentile is 219 ms, against 12 ms when the link is idle. Both numbers come out of the same simulation of the same link at the same moment. Two: switch the queue discipline to fq_codel, the fix everyone recommends, and change nothing else. The median stays at 187 ms. Not improved a little — unmoved, because the queue is not in the qdisc you just replaced. Three: leave fq_codel selected and drag the shaper from off to 98%. The median drops from 187 ms to 12 ms, which is to say the queue disappears entirely, and the speed test falls from 19.60 to 19.16 Mbps. That is the whole trade, and it is not the trade anybody thinks they are making.

The queue is yours, and the speed test is measuring the wrong thing

Nothing in the default panel is broken. The modem is doing exactly what a first-in-first-out buffer is supposed to do: hold packets it cannot send yet, rather than throwing them away. The sender is doing exactly what a loss-based congestion controller is supposed to do: grow its window until the path complains. The trouble is that those two correct behaviours compose into a system whose equilibrium is "the buffer is full."

Follow the loop once. CUBIC increases its congestion window every round trip. Above the bandwidth-delay product — the amount of data that fits in the path itself, rate times round-trip time — extra window does not become extra throughput, because the link is already busy every microsecond. It becomes queue. The window keeps growing anyway, because nothing has told the sender to stop. Only when the buffer is completely full does the modem discard a packet, and only then does the sender learn anything. So the steady-state window is the bandwidth-delay product plus the entire modem buffer, and the steady-state queue is whatever fraction of that buffer survives CUBIC's multiplicative decrease of 0.7. That is why the mean queue in the panel is 178 ms against the 210 ms the buffer can physically hold: CUBIC spends most of each cycle near the top of its window and only briefly below it.

Now the part that makes this invisible. Throughput is bytes served per second by the bottleneck link. The link is saturated the entire time. Adding queue does not add throughput and removing queue does not remove it — the panel reports 19.60 Mbps with a 178 ms standing queue and 19.16 Mbps with none at all. A speed test reports throughput. It is not that speed tests are inaccurate; they are accurate about a quantity that is unaffected. Some now report a separate latency-under-load figure precisely because of this, and that figure is the one that matters.

The delay is paid by every other flow behind the same buffer. Your upload is not slowed by its own queue: it is pipelined, so it keeps getting acknowledgements at line rate. The video call's 200-byte packet arriving at a full modem buffer has 512 KB of somebody else's bulk data in front of it and waits for all of it. Set the discipline to fq_codel and leave the shaper off, and the median stays at 187 ms — the call is not even queued at your router, so per-flow queueing at your router has nothing to isolate it from.

Why installing fq_codel changed nothing

A queue discipline can only manage a queue that forms inside it. A queue forms at the slowest hop, and only at the slowest hop. If your router hands packets to the modem faster than the modem can put them on the line, the router's queue is empty by construction, and the most sophisticated active queue management (AQM) algorithm ever written will spend the entire transfer looking at an empty queue and correctly concluding there is nothing to do. The log says this explicitly when you select fq_codel with the shaper off: it signalled zero times, because it never saw a packet that had waited 5 ms.

This is the single most common failure of a bufferbloat fix, and Linux's own defaults make it easy to hit. sch_cake, the modern successor to fq_codel, has a comment on the line that sets its rate: q->rate_bps = 0; /* unlimited by default */. Attach CAKE — Common Applications Kept Enhanced — to an interface without a bandwidth parameter and it will never become the bottleneck and never see the queue it was installed to manage. It will still show up in tc qdisc show, still report statistics, still look installed. Its drop counter will read zero and its latency target will never be exceeded, and people read that as success.

Shaping is what moves the bottleneck. A shaper is a deliberate rate limit on egress: your router refuses to hand packets to the modem faster than some number you choose. Choose a number below what the line can carry and your router becomes the slowest hop, so the queue forms in your qdisc, where CoDel — Controlled Delay — is watching.

The threshold is not a dial, it is a switch, and the panel makes that uncomfortable. With fq_codel selected, walk the shaper down from off. At 110%, 105% and 100% of the nominal line rate the median call latency is 188, 188 and 114 ms. At 98% it is 12 ms. There is no gradual improvement across that range because there is no gradual anything: either your router is the bottleneck or the modem is, and the queue is entirely in whichever one it is.

Why 100% is not low enough is worth pinning down, because it is where people give up. A 20 Mbps line does not carry 20 Mbps of payload. Outside every Ethernet frame the line pays 8 bytes of preamble and start-of-frame delimiter, 4 of frame check sequence and a 12-byte interframe gap; inside it, 14 more for destination, source and type. A 1500-byte IP packet therefore occupies 1538 bytes of line time, and a real 20 Mbps Ethernet link carries 19.51 Mbps of payload. The model charges only the 24 bytes outside the frame, so its readouts say 19.69 rather than 19.51 — a simplification that makes the panel's headroom slightly generous rather than slightly harsh, and the numbers below are the model's. Either way, a shaper configured for "20 Mbps" of payload is still faster than the line. On a real connection the gap is larger and less obvious — PPPoE adds 8 bytes, VLAN tags 4, DOCSIS its own framing, and ATM-based DSL pads every packet to a whole number of 53-byte cells, which can cost 10% on small packets. This is what tc qdisc ... cake bandwidth 19Mbit overhead 18 mpu 64 exists for, and why the working advice has always been to shape below the rate on the invoice rather than at it.

A smaller buffer is not the fix, and shaping alone can make it worse

The obvious response to "the buffer is too big" is to make the buffer smaller, and the panel will let you try. Drag the modem buffer down with everything else as it loads. 512 KB gives a 187 ms median call latency and 19.60 Mbps; 128 KB gives 54 ms and 19.60; 32 KB gives 19 ms and 19.59, still 98% of the line; 16 KB gives 13 ms and 19.00, which is 95%. The latency falls the whole way. The throughput holds and then, at the bottom, gives way — because a buffer's real job is to cover the gap between a multiplicative decrease and the window climbing back. When a sender halves — or in CUBIC's case multiplies by 0.7 — its window below the bandwidth-delay product, the link idles until the window recovers, and the buffer is what stops that from happening.

There is an arithmetic for where that knee sits. At the drop the window is one bandwidth-delay product plus the buffer; afterwards it is beta times that; for the link to stay busy it has to still be at least one bandwidth-delay product (BDP), which rearranges to buffer ≥ BDP × (1 − beta) / beta. For Reno's beta of 0.5 that is one whole bandwidth-delay product — the origin of the old "size the buffer at one BDP" rule — and for CUBIC's 0.7 it is 0.43 of one. The panel lands on that number without being told it. At 100 Mbps and the model's 24 ms round trip the bandwidth-delay product is 293 KB, so the prediction is 126 KB. Measured, at 100 Mbps with no shaper: 128 KB holds 98% of the line, 64 KB holds 94%, 32 KB holds 88% and 16 KB holds 72%. The knee is where the arithmetic says it is, and nothing in the model was told to put it there.

Read that arithmetic as an instruction and you have still lost, because it contains the round-trip time and the rate of the traffic, and a home router is told neither. Put the modem buffer back to 512 KB and walk the rate slider. The buffer readout by itself says 210 ms at 20 Mbps, 42 ms at 100 Mbps and 4.2 ms at 1 Gbps, and the median call latency reads 1,970 ms at 2 Mbps, 744 at 5, 370 at 10, 187 at 20, 79 at 50, 43 at 100, 21 at 250 and 13 at 1 Gbps. One buffer, eight answers, and it was never told which link it would end up behind. No single number is right.

Worse, shaping by itself can make the problem larger than doing nothing. Put the rate back to 20 Mbps with the buffer still at 512 KB, select pfifo_fast, set the shaper to 95%, and read the hero number: the median goes to 575 ms and the 95th percentile to 648 ms, against 187 and 219 with no shaper at all. You did exactly what the guides said — you moved the bottleneck onto your own router — and you moved it into a queue that is bigger than the one you escaped. Linux's default txqueuelen is 1000 packets; at 1514 bytes each that is 1.44 MB, nearly three times the modem's 512 KB. A first-in-first-out queue does not care how long anything has waited. It drops when it is full, and full is a number of packets, and the number of packets that constitutes half a second of delay depends on a rate the queue has never been told.

That is the actual content of the fix: not a smaller buffer but a queue that measures itself in time. CoDel timestamps every packet on the way in and looks at the sojourn time — how long the packet at the head of the queue has been sitting there — on the way out. If the minimum sojourn time stays above the 5 ms target continuously for the 100 ms interval, it drops the packet it just dequeued and schedules the next drop one interval later, then at interval divided by the square root of the number of drops so far, so the signalling rate rises smoothly until the standing queue is gone. RFC 8289 also forbids dropping from a queue holding one maximum transmission unit or less, which is what keeps it from destroying a link that simply has one packet in flight. None of those constants mention bytes, so none of them need to change when the rate does.

The flow queueing half is a separate idea and the panel separates them. Select codel — CoDel with a single shared queue — at a 95% shaper with 8 concurrent uploads: the call sees 20 ms at the median and 29 ms at the 95th percentile. Correct, and seven times better than the 221 ms the same eight uploads leave it with when the shaper is off, but it is still queued behind bulk data, so it inherits roughly the target sojourn time. Switch to fq_codel at the same settings, for the same throughput to two decimal places — 18.90 Mbps against 18.92 — and the call reads 12 ms and 12 ms, which is the idle latency, meaning the call is not waiting at all. fq_codel hashes each packet to one of 1024 queues and serves them by deficit round robin, and a flow with nothing backlogged is put on the new_flows list, which is served before old_flows. A sparse flow — a call, a DNS query, an SSH keystroke — is by definition always a new flow, so it goes out next rather than last. Neither half substitutes for the other, which is the point RFC 7567, the IETF's best current practice on queue management, makes in one line: "AQM is needed even for network devices that use per-flow or per-class queueing, because scheduling algorithms by themselves do not control the overall queue size or the sizes of individual queues." Flow queueing decides who goes next; CoDel decides how deep anyone is allowed to get.

Where the fix fails

A shaper is a constant. The rate of a real last-mile link is not. Put concurrent bulk uploads back to 1, keep fq_codel at a 95% shaper, and tick line rate wanders ±15%: the median goes from 12 ms to 17 ms and the 95th percentile from 12 ms to 47 ms. The mean queue sitting in the modem, which was 0.0 ms without the wobble, is 12 ms with it. Nothing about your configuration changed. For the part of each cycle when the line falls below the shaper, the bottleneck jumps back downstream and the call takes the hit. Drop the shaper to 90% and the 95th percentile is 37 ms; at 85% it is 18 ms, costing 3.0 Mbps of the 19.69 the line can carry; at 80% it is 12 ms flat and you are paying 4.0 Mbps for it.

That is the real cost of static shaping, and it is why the shaper has to be set below the minimum rate the link will reach rather than its average. On a DOCSIS cable segment the usable rate falls when the neighbourhood is busy. On DSL it changes with line conditions. On LTE and 5G it changes constantly. On Wi-Fi between your device and the router it changes with every rate adaptation, and there is a second unmanaged queue in the access point that your router's egress qdisc cannot see either. Ninety comments on cake-autorate issue 350 are ninety comments about exactly this: setting one number below a line rate that will not hold still. The tools that solve it — cake-autorate, CAKE's own ingress mode — work by measuring latency continuously and moving the shaper, which is a control loop wrapped around a control loop.

Three more boundaries. The first is the one the panel cannot show you, because it only models egress. Everything above is the upload direction, where you control the sending device. For download, the queue is in your provider's equipment and the packets have already crossed it by the time you see them; a shaper on ingress works only by dropping traffic that already consumed the bottleneck, which is why ingress shaping needs more headroom than egress and why CAKE has a separate ingress keyword that accounts for it.

Second, speed. Untick the wobble again, move the rate slider to 1 Gbps, and go back to the default 512 KB buffer with no shaper and pfifo_fast: the median call latency is 13 ms and the 95th percentile 15 ms, against 12 ms idle, with the speed test at 936.0 Mbps. Bufferbloat evaporated and nobody fixed anything. 512 KB at 1 Gbps is only 4.2 ms of transmission time, so there is no room in it for a standing queue worth noticing. This is why the problem concentrates on slow uplinks, and why it is so often reported as having started when somebody moved house. It cuts hard the other way: at 2 Mbps that same 512 KB is 2.1 seconds of buffer, and the panel measures a 1,970 ms median — two full seconds of one-way delay on a link that is passing 94% of its rated throughput.

Third, the sender. Everything here assumes a loss-based controller that has to fill the buffer to find its limit. BBR — Bottleneck Bandwidth and Round-trip propagation time — estimates both of those directly and paces to their product, so it does not need the buffer to tell it when to stop — but you only control the senders you run, and the buffer is still there for everyone else. ECN removes the retransmission from the signal: with Explicit Congestion Notification negotiated, CoDel marks the packet instead of discarding it, and the sender reduces its window just the same. Look at cobalt_should_drop in sch_cake.c and that is literally the order of preference: if (!(vars->ecn_marked = INET_ECN_set_ce(skb))) reason = QDISC_DROP_CONGESTED; — mark if you can, drop if you cannot.

Checking it on a real system

Never measure this with an idle ping. An idle ping measures the one thing bufferbloat does not affect, for the same reason a speed test does. The measurement is the difference between loaded and unloaded latency, so you have to load the link and ping at the same time:

• In one terminal, saturate the direction you suspect: iperf3 -c host -t 60 for upload, -R for download. In another, ping -i 0.2 1.1.1.1. The number is the loaded median minus the idle median. Under 30 ms of added delay is fine; over 150 ms and interactive traffic is unusable.
• ss -ti on the bulk socket gives you the same answer from the sender's side in one line. minrtt: is the lowest round-trip time ever observed on that socket and rtt: is the smoothed current one. The gap between them is the standing queue. minrtt:12 rtt:190/3 is 178 ms of bufferbloat, stated as plainly as any diagnostic ever states anything.
• tc -s qdisc show dev eth0 is how you find out whether your qdisc is doing anything. For fq_codel look at maxpacket, drop_overlimit, new_flows_len and ecn_mark. A CoDel-family qdisc with zero drops and zero marks under a saturated link has not fixed your latency; it has proved it is not the bottleneck. For CAKE, a bandwidth shown as unlimited means the same thing.
• cat /proc/sys/net/core/default_qdisc, and ip link show dev eth0 | grep qlen. Do not assume this says fq_codel. The kernel's own default is still pfifo_fast — net/sched/sch_generic.c has default_qdisc_ops = &pfifo_fast_ops and CONFIG_DEFAULT_NET_SCH falls back to "pfifo_fast". What puts fq_codel on most Linux machines is systemd, whose sysctl.d/50-default.conf carries net.core.default_qdisc = fq_codel under the comment "Fair Queue CoDel packet scheduler to fight bufferbloat". A minimal container image, an appliance distribution or an initramfs without it gets pfifo_fast and qlen 1000.

The fix, in the order it has to be done. Put the queue on a device you control: tc qdisc replace dev eth0 root cake bandwidth 19Mbit overhead 18, with a bandwidth strictly below the rate the line delivers, not the rate on the invoice, and an overhead that matches the link technology. Then re-run the loaded ping. If the added delay did not move, the bottleneck is still downstream — check tc -s qdisc for a drop counter that is still zero, and walk the shaper rate down until it is not. If the added delay is fixed but throughput fell more than you can accept, raise the shaper toward the line rate one step at a time; the last step before the latency comes back is your answer, and it will need re-checking whenever the line rate changes.

On a Linux server rather than a router the question is different and usually smaller. The host is rarely the bottleneck, so its own qdisc rarely queues. What does matter there is that fq — not fq_codel — is the qdisc that implements pacing for BBR, and that if you have set txqueuelen to something large on a virtual interface you have created a bufferbloat machine on your own host. The still-open argument on systemd issue 9725 over whether that shipped default should be fq instead of fq_codel is, underneath, an argument about who owns the queue: a host that is never the bottleneck wants a pacer, and a host that might be wants an AQM.

Next: the ramp that decides where the window starts, slow start; the other way a link with spare capacity delivers terrible latency, Nagle and delayed ACK; the marking scheme that makes the congestion signal free, ECN; and the sender that estimates the pipe instead of overflowing it, BBR.