The KV Cache

Generating a token does almost no arithmetic. It reads a very large amount of memory and then does almost no arithmetic. That single fact decides how many users your GPU can serve, and it is why a card with three times the FLOPs can hand you no extra tokens per second at all.

A decoder-only transformer never recomputes the past. At every layer it projects each token once into a key and a value, and keeps them. When the next token arrives, its query attends over every key and value produced so far — see scaled dot-product attention for the step itself. Those retained tensors are the KV cache, and their size is fixed by the architecture:

bytes per token = 2 (K and V) × layers × kv_heads × head_dim × bytes_per_element

kv_heads, not query heads. That distinction is the whole of grouped-query attention and it is worth a factor of eight on a modern model. Multiply by context length, multiply again by the number of concurrent sequences, and you have the number that actually sets your concurrency limit.

The panel below is a serving engine's admission decision and its per-step memory traffic. Pick a model and a GPU, then push the prompt length up and watch what stops you.

Start at the defaults and drag prompt tokens from 4k to 32k. Nothing about the model changed and the GPU still has the same FLOPs, but the number of sequences the engine can hold falls from 93 to 12 and throughput follows it down, 2,300 tokens per second to 430. Now set kv cache precision to FP8: the bytes moved per decode step do not change at all — the step still takes 27.9 ms — yet throughput doubles, because the same traffic is now serving 24 sequences instead of 12. That is the shape of the whole problem — you are not buying arithmetic, you are buying sequences per byte.

Why the batch stops growing where it does

A serving engine does not decide concurrency from a config value. It fills HBM in a fixed order and whatever is left becomes the KV pool:

1. Take gpu_memory_utilization × capacity. The remainder is for the CUDA context, the allocator and the driver.
2. Load the weights. Fixed cost, paid once, independent of load.
3. Reserve activation and workspace room for the largest forward pass the scheduler is allowed to launch.
4. Everything still free is the KV pool. Concurrency is floor(pool ÷ bytes_per_sequence), and nothing else.

Put kv cache precision back to BF16 first — the paragraph above left it on FP8, and every byte figure below is a BF16 one — then set the model to Qwen3-32B and watch step 2 eat the card. The weights alone are 61 GiB of a nominal 80 GiB; at gpu_memory_utilization = 0.90 there are about 10 GiB left, and a 32k-token request wants 8 GiB of that. You can serve one user. Drop gpu_memory_utilization to 0.72 and the log tells you the weights no longer fit at all — which is the failure mode behind every "it worked at 0.9 and OOMed at 0.85" bug report.

This is also why turning max_num_seqs down "to fix OOM" so often makes throughput worse rather than better. The pool was never the problem the knob addresses. It caps how many sequences the scheduler will try to run; the pool caps how many it can. Lowering the knob below the pool limit gives up batching you had already paid for, and — see the next section — batching is the only thing amortising the weight read.

The simulation gives each sequence a contiguous block sized for its full final length. Real engines do not: paged attention allocates the cache in fixed-size blocks so a sequence only holds what it has actually generated, and continuous batching admits a new request the moment another finishes. Both of those raise the achievable batch, sometimes by a lot. Neither changes the arithmetic above — they just stop you wasting the pool.

Two phases, opposite bottlenecks

Look at the second panel with a long prompt and a short generation — say 32k in, 64 out, which is roughly the shape of a retrieval-augmented request.

Prefill processes the entire prompt in one forward pass. Every prompt token goes through every matmul simultaneously, so the weights are read once and reused thousands of times. Arithmetic intensity is high, the tensor cores are the constraint, and time to first token scales with prompt length — and with prompt length squared in the attention term, once the prompt is long enough for that term to matter. More FLOPs genuinely help here.

Decode produces exactly one token per sequence per step. The same weight matrices are read from HBM again, and this time each one is used for a single vector per sequence. Add the KV cache, which must be read in full because the new query attends over all of it. The step is a memory transfer with a rounding error of arithmetic attached. Shazeer's multi-query attention paper named this in 2019: incremental inference is slow "due to the memory-bandwidth cost of repeatedly loading the large keys and values tensors."

Neither label is a property of the phase; both are ratios, and the panel will flip them for you. Put the model back to Qwen3-8B and gpu_memory_utilization back to 0.90 — the section above left the panel in the state where the weights fill the card and there is no forward pass to measure — then drag prompt tokens down to 128 and prefill goes bandwidth-bound: 128 tokens are not enough work to pay for reading 15 GiB of weights, so a short prompt costs almost exactly what an empty one does — which is why prompt length below a few hundred tokens barely moves TTFT. Push the other way — 128-token prompts, 16 tokens out, 256 concurrent on Qwen3-14B — and decode goes compute-bound: the caches are tiny, the weight read is 83% of the step and is split 256 ways, and the batched matmul finally has enough columns to saturate the tensor cores. That is the real statement. Decode is bandwidth-bound at the batch sizes and context lengths anyone actually serves, not by definition.

The consequence people miss is that time to first token and inter-token latency have no lever in common. Compare the A100 and the H100 on Qwen3-14B at 4k, 64 concurrent: 3.17× the FLOPs cuts TTFT by exactly that factor, 745 ms to 235 ms, while ITL improves only in proportion to bandwidth, 46 ms to 28 ms — 1.64×. Now switch to the H200: identical compute, so TTFT does not move by a millisecond, while ITL drops again and the pool more than doubles. If your users complain about the wait before text appears, you have a prefill problem and should buy compute. If they complain that text crawls, you have a bandwidth problem and compute will not touch it.

Because the two phases stress different units, a slice of prefill folded into a decode step rides on a weight read that step was making anyway, and spends tensor cores the readout was just showing you at a tenth of duty. It is close to free right up to the point where the added arithmetic exceeds the step's memory time, which is exactly what a token budget like max_num_batched_tokens is sizing. Run them in separate steps instead and one long prefill stalls every decode behind it. That is the entire argument for chunked prefill. It is also why speculative decoding works at all: decode has idle FLOPs, so verifying several candidate tokens in one pass costs almost nothing beyond the pass you were already making.

What GQA buys, and where it stops helping

Turn on force multi-head attention. On Qwen3-8B the cache goes from 144 KiB per token to 576 KiB, because kv_heads went from 8 to 32 and it is a linear term; resident sequences at 4k context fall from 93 to 22. The weights grow too — 8.19B parameters to 9.10B, since k_proj and v_proj widened — but an 11% rise in the fixed cost against a fourfold rise in the per-token cost is not a close contest. On Qwen3-32B, where the ratio is 64 query heads to 8 KV heads, the cache goes from 256 KiB per token to 2 MiB and the model can no longer hold a single 4k request on an 80 GiB card.

Now read the throughput number carefully, because the reason it fell is not the obvious one. The bytes moved per decode step are almost identical: the KV pool is the same size and the engine fills it either way. What changed is how many sequences that traffic serves. Under GQA the same read produces eight times as many tokens. Bandwidth-bound throughput is sequences per byte of cache, and the KV head count is the cheapest place to buy them.

Two mitigations that look equivalent are not:

• FP8 KV cache halves bytes per token. When the pool is what is turning requests away it doubles resident sequences and doubles throughput while the step time barely changes. When the pool is not full it buys nothing at all — the same 32 sequences move less cache, so the step gets slightly faster and that is the end of it.
• FP8 weights halve the fixed cost, which enlarges the pool — a big win on a model that nearly fills the card, and almost nothing on a small one. On Qwen3-8B at 4k context the pool goes from 55.7 GiB to 63.3 GiB and you gain about a dozen sequences. On Qwen3-32B it goes from 9.8 GiB to 40.3 GiB and you gain four times as many.

Which one to reach for depends entirely on whether weights or cache dominate your budget bar, and quantization has accuracy costs that differ between the two. Read the bar before choosing.

The boundary. Set Qwen3-8B, 32k prompt, 256 concurrent requests, FP8 everywhere, then compare it against the same thing at 8 concurrent requests. The batch went up 3.4× — 8 sequences to 27 — and bought 1.26× the aggregate tokens per second, while every individual stream got 2.7× slower. Watch the log line about the weight share to see why: at batch 8 the weight read is 30% of the step, so each extra sequence partly rides along free. At batch 27 it is 11%, and there is nothing left to amortise. Every additional sequence now adds its own full cache to every token, ITL rises roughly linearly with the batch, and aggregate throughput flattens. Batching is amortisation, and amortisation has a ceiling.

This is the ceiling that tensor parallelism is really for. Shard across four GPUs and each rank reads a quarter of the weights and a quarter of every sequence's cache, so a decode step moves a quarter of the bytes: inter-token latency and aggregate throughput both improve, and the pool grows by more than four times because the weights stop being paid for four times over. It is bought with an all-reduce on every layer, which is why the scaling is sublinear and why tensor parallelism wants an NVLink domain rather than a network. The ceiling it does not move is the kv head count: you cannot split 8 KV heads across more than 8 ranks without replicating them.

One thing that does not help: FlashAttention is frequently assumed to shrink the cache. It does not. It removes the materialised N × N score matrix and the HBM round-trips that go with it — an activation-memory and prefill win. The keys and values still exist, still occupy the pool, and are still read in full on every decode step.

Checking it on a real system

Do the arithmetic before you touch a knob. From the model's config.json: 2 × num_hidden_layers × num_key_value_heads × head_dim × bytes. Qwen3-8B at BF16 is 2 × 36 × 8 × 128 × 2 = 147,456 bytes — 144 KiB per token, 1.13 GiB for an 8k-token request. If head_dim is absent, it is hidden_size ÷ num_attention_heads, but check: Qwen3-32B sets it explicitly to 128 while hidden_size ÷ num_attention_heads is 80.

Then read what the engine already told you at boot. vLLM prints the pool it carved out and what that means for you, in one line:

GPU KV cache size: 1,048,576 tokens, Maximum concurrency
for 32,768 tokens per request: 32.00x

That second number is your concurrency ceiling before a single request arrives, and it is computed from exactly the arithmetic above. Compare it against your target QPS × average request duration. If it falls short, no amount of tuning max_num_seqs will help — the knob is above the ceiling, not below it. Note that vLLM divides by the model's maximum context, so if your real p95 prompt is a tenth of that, your real ceiling is ten times what the line says.

Under load, the steady-state line is the diagnosis. It reads Avg generation throughput: … Running: N reqs, Waiting: M reqs, GPU KV cache usage: X%, Prefix cache hit rate: Y%, and it splices an extra Preemptions: P field in ahead of the cache-usage field, but only once preemptions have actually happened — so the field's mere presence is the alert. Cache usage pinned near 100% with a non-empty Waiting queue means the pool is your binding constraint. A rising Preemptions count means worse: the scheduler admitted work it could not hold, and is now discarding completed prefill and recomputing it, so you are paying for the same prompt twice. The Prometheus equivalents are the vllm:kv_cache_usage_perc gauge and the counter vLLM registers as vllm:num_preemptions and the client library exports as vllm:num_preemptions_total. Alert on the second one; it should be zero.

To confirm the bandwidth diagnosis on the hardware itself, sample DCGM's DCGM_FI_PROF_DRAM_ACTIVE and DCGM_FI_PROF_PIPE_TENSOR_ACTIVE during steady-state generation. Decode-heavy traffic shows DRAM activity high while tensor-pipe activity sits in the low single-digit percent. That combination — busy memory, idle tensor cores, and nvidia-smi cheerfully reporting near-100% "utilization", which only means a kernel is resident — is the picture this lesson exists to explain.

Next: the architecture that made long contexts affordable, grouped-query attention; the allocator that stopped the pool being wasted, paged attention; and the reason the same prompt costs nothing the second time, prefix caching. If you want the general form of the bandwidth-versus-FLOPs argument, it is arithmetic intensity.