LLM internals / inference / performance / hardware
Arithmetic Intensity and the Roofline
That memory-bound and compute-bound are labels attached to operations, so attention is 'the memory-bound one' and matmuls are 'the compute-bound ones'. Intensity is a property of a specific execution, not of an operation: the same weight matmul runs at 1 FLOP per byte with one token in the batch and 700 with a thousand, which is the entire reason batching works. Two consequences people miss. Batching moves the weight matmuls along the roofline but cannot move attention over the KV cache at all, because each sequence's cache is read by exactly its own query, so a big batch at long context is still memory-bound and the throughput gain stops arriving. And the ridge point has been rising with each GPU generation - 153 on an A100, 295 on an H100 - so upgrading hardware makes more of your kernels memory-bound, not fewer.
An H100 can perform 295 floating-point operations in the time it takes to fetch one byte from its own memory. A transformer generating a token for one user performs about 1.2. The tensor cores you paid for are idle 99.7% of the time, and no amount of extra arithmetic capability changes that number.
Arithmetic intensity is the ratio you get by dividing the floating-point operations a piece of work performs by the bytes it has to move out of memory to perform them. It is measured in FLOPs per byte. Every processor has its own version of the same ratio — peak FLOP/s divided by peak memory bandwidth — and comparing the two tells you which resource the work is waiting on. That comparison is the roofline model, from Williams, Waterman and Patterson in 2009, and it says something almost embarrassingly simple:
attainable FLOP/s = min( peak FLOP/s , peak bandwidth × arithmetic intensity )
The intensity where those two terms are equal is the ridge point. Below it you are memory-bound and the tensor cores wait; above it you are compute-bound and the memory system waits. An A100's ridge point is 312 TFLOP/s ÷ 2.04 TB/s = 153 FLOPs per byte. An H100's is 989 ÷ 3.35 = 295.
Here is one layer of Qwen3-8B, kernel by kernel, with every FLOP and every byte counted. The control that carries this lesson is tokens in this forward pass. Start at 1 and drag it right.
Both phases run the same matrix multiplies over the same weights; the only structural difference is where the tokens came from and what attention has to read. That is why one slider serves both — a batch of 64 decodes and a 64-token prefill chunk put the same 64 rows through every weight matrix.
The sloped part of the roof is the memory system: nothing there can go faster than bandwidth × intensity. The flat part is the tensor cores. Each dot is one kernel, placed at its intensity; the dot's size is its share of the pass, and marks the kernel taking the most time. A dot sitting on the roof is at its ceiling, not necessarily at its measured speed.
Qwen3-8B's shape is from its published config.json: 36
layers, hidden size 4096, 32 query heads and 8 kv heads of 128, MLP width
12,288, vocabulary 151,936. FLOP and byte counts are exact arithmetic for
those shapes — weight bytes, activation bytes, cache bytes, all of it.
Times apply 80% of quoted bandwidth and 75% of quoted dense BF16
tensor-core rate, and add the kernels up as if they run one after another
with no overlap, so treat them as a well-shaped estimate rather than a
benchmark. The roofline itself uses the unadjusted hardware peaks, which
is what a roofline is for. Prefill attention is modelled as a tiled kernel
that reads K and V once, which slightly understates its traffic.
At one token the whole pass runs at 1.22 FLOPs per byte and reaches 0.3% of what the H100 can do — 164 tokens per second, and 15.2 GiB moved to produce 20 GFLOP of arithmetic. Drag to 64 tokens and the weight matmuls climb to 62 FLOPs per byte, still short of the ridge point but sixty times better, and throughput goes from 164 to 1,840 tokens per second. That is the entire economic argument for batching, and it is a statement about intensity rather than about parallelism.
Then drag to 256 and watch it stop working. Four times the batch buys 14% more throughput.
Batching walks you up the slope, and then stops
Set the pass to 64 tokens with 8k of context and read the ledger. The weight matmuls are at 62 FLOPs per byte and take 15% of the time between them. Attention takes 83%. Now go to 256 tokens: the matmuls climb to 232 FLOPs per byte — nearly at the ridge point — and attention takes 94.5% of the pass. Throughput went from 1,840 tokens per second to 2,096. You quadrupled the batch and bought 14%.
The reason is one number that never moves. A weight matrix is shared: read it once, use it for every token in the pass, and its intensity rises exactly in proportion to how many tokens rode along. A KV cache is private: sequence 200's query reads sequence 200's keys and nothing else. Adding a sequence adds both its FLOPs and its bytes, in the same ratio, forever. That ratio is
attention FLOPs per cache byte = 2 × query heads ÷ ( kv heads × bytes per element )
which for Qwen3-8B at BF16 is 2 × 32 ÷ (8 × 2) = 4.00, and the ledger will show you 4.00 at every batch size and every context length you can select. Four, against a ridge point of 295. The attention kernel of a decode step is seventy times too memory-hungry for the machine it runs on, and it is the only kernel in the model that cannot be fixed by scheduling.
Two things do move it, and both are architectural rather than operational. Switch kv cache dtype to FP8 and the intensity doubles to 8.00, because the same arithmetic now reads half the bytes. And the ratio's first term is exactly what grouped-query attention is: multi-head attention would put it at 1.00 and multi-query at 32.00. That is the same fact the KV cache lesson states in bytes, restated in FLOPs per byte, and it is why the two levers people reach for — more batch, faster GPU — both fail on this one kernel.
The escape is to make the context shorter, not the batch bigger. Set context to 512 and batch to 256: attention's share drops to 52%, and throughput goes from 2,096 tokens per second to 18,319. Nothing about the hardware changed. The same batch over a shorter context is a different point on the roofline.
The ridge point is moving away from you
Compare the A100 and the H100 at one token of decode. The H100 has 3.17× the tensor-core throughput and 1.64× the bandwidth. Single-stream generation goes from 100 tokens per second to 164 — 1.64×, the bandwidth ratio, to the digit. Every one of those extra FLOP/s went unused.
This is not an accident of one generation. Peak arithmetic has grown faster than memory bandwidth for twenty years, so the ridge point rises with every product: 153 FLOPs per byte on an A100, 295 on an H100. The H200 is the interesting case, and it is in the control set — same compute as the H100, 1.43× the bandwidth, so its ridge point falls to 206. It is the only one of the three that moves the boundary in the direction inference wants.
The practical consequence is that a kernel which was comfortably compute-bound on one generation can be memory-bound on the next without a line of code changing. Set the pass to 128 tokens of prefill on an A100: the matmuls sit at 122 FLOPs per byte, below that GPU's ridge of 153, and the whole prefill is memory-bound — which is why a short prompt costs almost the same as an empty one. The crossover for prefill on an H100 is around 300 tokens. Prefill is compute-bound at the prompt lengths people actually serve, but it is not compute-bound by definition, and the boundary is a number you can compute rather than a property you can assume.
Quantization is an intensity trick, and this is how to predict what it will do for you. Switch weight dtype to FP8 at one token: 164 tokens per second becomes 305. INT4: 536. Nothing about the arithmetic changed — the FLOPs are identical, the model runs the same multiplies — but the bytes halved and halved again, so the intensity doubled and doubled and the bandwidth-bound kernels finished proportionally sooner. Now set the pass to 1,024 tokens with 512 of context, where the matmuls are already compute-bound, and switch dtype again. The throughput readout does not move at all — 19,609 tokens per second at BF16, at FP8 and at INT4 — because you are shrinking a resource that stopped being the constraint. In a real system it is slightly worse than "no faster": an INT4 kernel has to unpack its weights before multiplying, which is arithmetic this model does not charge for and which lands squarely on the side that is already busy. That is the whole reason weight-only quantization is celebrated for single-stream latency and quietly disappointing in high-throughput serving, and the roofline predicts it before you install anything.
The kernels nobody thinks about tell the same story from the other end. Look at the row for norms, SiLU and residual adds: 0.55 FLOPs per byte, always memory-bound, and at a 4,096-token prefill it is 9.7% of the pass — more than the attention it sits next to. It is doing almost no arithmetic; it is just touching every activation. This is the reason kernel fusion exists, and it is invisible to anyone reasoning about FLOPs alone.
Checking it on a real system
Do the arithmetic first, because it takes one minute and usually ends the investigation. For decode with a batch of B, the bytes are the whole weight set plus the resident cache, and the FLOPs are twice the parameter count per token:
bytes = params × bytes_per_weight + B × ctx × 2 × layers × kv_heads × head_dim × kv_bytes
FLOPs = 2 × params × B + 4 × layers × q_heads × head_dim × ctx × B
intensity = FLOPs / bytes
Compare that against your GPU's peak FLOP/s divided by its bandwidth. If
your intensity is below the ridge point, your ceiling is
bytes ÷ bandwidth seconds per step and nothing on the compute
side can help. For an 8B model at BF16 with a small batch, the ceiling is
roughly 16 GB ÷ 3.35 TB/s ≈ 4.8 ms per token, about 200 tokens
per second, and that is the number to compare your measurement against
before concluding anything is wrong.
Then measure two counters, not one.
nvidia-smi's "GPU-Util" is the fraction of time at least one
kernel was resident. It says 100% for a kernel that is doing nothing but
waiting on memory, which is why it is the single most misleading number in
this stack. Use DCGM instead:
DCGM_FI_PROF_DRAM_ACTIVE # fraction of peak memory bandwidth in use
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE # fraction of peak tensor-core issue in useHigh DRAM, low tensor: memory-bound, and the fixes are fewer bytes — a smaller dtype, fewer kv heads, shorter context, or more aggregate bandwidth. Low DRAM, high tensor: compute-bound, and the fixes are fewer FLOPs or a bigger GPU. Both low: you are latency-bound on something else entirely — kernel launches, Python overhead, a collective, or a scheduler that is not keeping the GPU fed. That third case is common and is the one the roofline does not describe.
For a per-kernel breakdown, Nsight Compute reports "Memory Throughput" and "Compute (SM) Throughput" as percentages of peak for each kernel, which is the roofline restated one kernel at a time; the tool will name the kernel that is costing you and which axis it is stuck on. Serving stacks report the same idea at a coarser grain as MFU — Model FLOPs Utilization, the fraction of peak FLOP/s your useful arithmetic achieved. Training runs routinely reach 40–60% MFU. Decode-heavy inference reaching 5% is not a bug report; it is the roofline, and the number to improve is tokens per second per dollar, not MFU.
One caution about the model in this lesson: it adds the kernels up as though they run one after another. Real kernels overlap a little, and a well-tuned stack fuses the small memory-bound ones into their neighbours, so measured times come in under the sum. The ratios and the intensities are exact; the milliseconds are a sketch. If your measurement is within a factor of two of this and on the same side of the ridge point, the model has told you what you needed.
Your decode workload runs at 4% of peak FLOP/s on an H100. Someone proposes moving to a GPU with twice the tensor-core throughput and the same memory bandwidth. Context lengths and batch size stay the same. What happens to tokens per second?
Next: what all those bytes actually are, the KV cache; the architectural lever on the one kernel batching cannot help, grouped-query attention; a kernel that adds arithmetic to save bytes, FlashAttention; and the scheduling decision that puts compute-bound and memory-bound work in the same pass on purpose, chunked prefill. The same trap in a different stack — a utilisation number that reads busy while the thing you care about is stalled — is CPU throttling under CFS quota.