What Quantization Actually Changes

Quantization is not one decision. It is four — the weights, the activations, the KV cache and the accumulator the arithmetic lands in — and choosing a bit width settles only half of any of them. The other half is how many numbers are forced to share a scale factor, and that is what decides whether the low bits land anywhere useful.

Start with the one everybody means. Weight-only quantization stores each weight in fewer bits and converts it back to 16-bit floating point inside the kernel, just before the multiply. Nothing else about the model changes: the activations arriving at the layer are still 16-bit, the matmul still runs on the 16-bit tensor cores, and the KV cache — the keys and values kept for every past token — is untouched. What you bought is bytes read from memory, and nothing else.

A quantized weight is not stored alone. It is stored as a small integer plus a scale — the real number that one integer step is worth — and usually a zero point, the integer that represents zero. A group of weights shares one scale, and the scale has to cover the largest magnitude in the group, because anything above it is clipped. So the biggest weight in a group sets the step size for every other weight in it. That sentence is the entire lesson of the first simulation.

Below is one linear layer — 32 output channels, 256 input channels — being quantized for real. The activations it is fed have three outlier channels: dimensions whose values are far larger than the rest, which is a documented property of transformers above about 6.7 billion parameters, not a fiction. Set the weight format to INT4 and then move the granularity control from per-tensor down to group 32.

Per-tensor INT4 gives the median group 16 usable levels spread across the largest weight in the entire matrix, and the layer's output error is 27.73%. Per-output-channel, which is the weakest granularity any real LLM toolchain ships, cuts that to 14.16%. Group 128 — what Qwen3-8B-AWQ actually uses — reaches 12.76%, and group 32 reaches 10.16%. Now switch the weight format to INT8 and run the same ladder: 1.61%, 0.846%, 0.718%, 0.489%. Seventeen points of error hang on the granularity control at four bits; one point hangs on it at eight. That is the first thing the single "bit width" dial hides.

Where the bytes actually go

Take the published Qwen3-8B-AWQ checkpoint. Its config.json says bits: 4, group_size: 128, zero_point: true. Four bits, so a quarter the size of the BF16 original — 16.38 GB should become 4.10 GB.

It is 6.10 GB. Here is every byte of the difference, and none of it is surprising once you count it.

Qwen3-8B has 8,190,735,360 parameters. Of those, 6,945,767,424 sit in the linear layers inside the transformer blocks, and those are the only ones AWQ touches. The remaining 1,244,967,936 — a 151,936 × 4,096 embedding table, an identically sized untied language-model head, and the layer norms — stay in BF16, because quantizing an embedding lookup buys nothing and quantizing the output head costs measurable quality for 7% of the weights. That is 2.49 GB that never moves.

Then the four bits are not four bits. Each group of 128 weights carries a 16-bit scale and a 4-bit zero point, so the real cost is:

4 + 16/128 + 4/128 = 4.15625 bits per weight

Multiply out: 6,945,767,424 × 4.15625 ÷ 8 = 3,608,543,232 bytes of quantized linear layers, plus 2,489,935,872 bytes of BF16 leftovers, is 6,098,479,104 bytes. The published checkpoint's safetensors files total 6,098,581,864 bytes. The 102,760-byte gap is the JSON header. The same arithmetic on Qwen3-32B-AWQ predicts 19,325,298,688 against a published 19,325,481,744, and on Qwen3-8B-FP8 — which uses 128 × 128 blocks of BF16 scales rather than group-wise ones — 9,436,551,168 against 9,436,628,784.

So INT4 group 128 is a 2.69x reduction, not 4x. Group 32 would be 4.625 bits and 2.52x. This is the answer to "why is my 4-bit model bigger than a quarter of the original", and it is also why the effective-bits readout in the first simulation is worth watching. Set that simulation's weight format back to INT4 — the INT8 ladder above left it at eight bits — and step group 128 against group 32. The effective-bits readout goes 4.156 to 4.625: 0.625 bits of metadata per weight, more than an eighth of the payload, to buy an accuracy step from 12.76% to 10.16% in that toy layer.

Fewer bytes only buys time where bytes were the constraint

A decode step reads every weight in the model and every byte of the KV cache, and does one token's worth of arithmetic with them. It is memory-bound by an enormous margin. A prefill pass reads the same weights once and does thousands of tokens' worth of arithmetic with them, and is compute-bound. Weight-only quantization changes the numerator of that ratio and nothing else, so it is worth close to the byte ratio in one case and worth zero in the other.

The panel below runs one decode step and prices six real serving configurations at once. Move sequences in the batch from 1 to the right-hand end and watch the ranking invert.

At batch 1 on an H100, BF16 Qwen3-8B moves 15.74 GB per step and takes 5.87 ms; INT4 group 128 moves 5.46 GB and takes 2.04 ms. That is 2.88x, and it is entirely a bandwidth result — the arithmetic in both cases takes 0.03 ms, one part in two hundred. The tensor cores are idle either way. Set readouts describe to BF16 weights to see that split: 14.1 GiB of weights against 576 MiB of cache. On INT4 the weight half is 4.52 GiB and the cache half has not moved.

Now set the context to 128 and the batch to 1,024 — short prompts, heavy concurrency, which is what classification and extraction traffic looks like. The BF16 step is compute-bound at 28.6 ms. The INT4 step is compute-bound at 30.1 ms, because the dequantized matmul runs on the same BF16 tensor cores as the unquantized one and the dequantization is not quite free. The speedup readout says 0.95x. Set weight-only kernel quality to the generic kernel and it says 0.55x — the quantized model is now half the speed of the one it replaced. That is not a bug report waiting to happen; it is several bug reports that have already happened, and vLLM prints a warning about it at startup that most people scroll past: "quantization is not fully optimized yet. The speed can be slower than non-quantized models."

Put the kernel back to fused dequant, Marlin class before reading anything else — the generic setting is a diagnosis, not a place to stand, and it holds the INT4 row at 0.55x for the rest of the page. The two W8A8 rows behave in the opposite way, and for the opposite reason. Quantizing the activations as well means the multiply itself happens in 8 bits, and an H100's INT8 and FP8 tensor cores run at 1,979 TOPS against 989 TFLOP/s for BF16. So W8A8 halves the compute-bound region as well as halving the bytes: at this setting it is 1.99x where INT4 is 0.95x. Four bits on the weights beats eight bits on everything by 2.88x to 1.79x at batch 1 and 4,096 tokens of context — put both back to check it, because at the 128-token context you are standing in it reads 3.11x to 1.85x — and loses to it by 0.95x to 1.99x at batch 1,024 with those short contexts. The crossover is somewhere in your traffic mix, and a single "bit width" dial cannot express which side of it you are on.

There is a third regime, and it is the one most people are actually in. Set the context back to 4,096 and leave the batch at 1,024. On an H100 the BF16 model admits 97 sequences and the INT4 model admits 114 — and at those batch sizes the cache is 64.1 GiB of the 68.6 GiB moved per step. The weights are now 7% of the traffic, and shrinking them further buys 1.18x. Switch the KV cache dtype to FP8 and, with the weights left in BF16, the same card admits 194 sequences and returns 2.00x. Do both and it is 2.35x. At this operating point the cache dtype was worth more than the weight dtype on its own, and it is a different flag, in a different section of the documentation, with a different accuracy argument behind it.

Three methods, one piece of algebra

GPTQ, AWQ and SmoothQuant get discussed as competitors. Two of them are literally the same transformation, and the third is doing something else entirely.

SmoothQuant and AWQ both pick a per-input-channel factor s, divide the activations by it and multiply the weights by it. The layer's output is unchanged, because (x/s)·(Ws) = x·W. All that moves is which side of the multiply has the awkward dynamic range. SmoothQuant sets s_j = sqrt(max|x_j| / max|w_j|); AWQ sets s_j = sqrt(mean|x_j|). Both are large on the loud channels, so both scale those channels' weights up. They point the same way, which is the thing most often got wrong about them — they are not opposites. What differs is the objective. SmoothQuant is dividing the activations down onto a grid an even 8-bit integer format can actually represent, and its abstract says what that costs: it works by offline migrating the quantization difficulty from activations to weights. AWQ is not trying to fix the activations at all. It is buying finer weight steps for the channels the output is mostly made of, and paying with coarser steps everywhere else. Same vector, different reason to want it, and you can only have one of them.

You can watch the migration happen, but you have to watch the right readout. Set the first simulation to INT4, group 128, 16-bit activations, outlier magnitude 20x, all three toggles off — one step up from the group 32 the byte section left you at. Now tick SmoothQuant and read the weight tensor error, which is the layer's weights compared with the originals and is not weighted by anything: it goes from 11.60% to 17.33%, and the decision log reports the spread of per-input-channel weight magnitude widening from 8.9x to 12.8x. Push the outlier magnitude to 100x and the weight error goes to 31.99% against a spread of 28.7x. That is the sentence from the abstract, as a number: the more difficulty there is in the activations, the more of it lands in the weights.

The layer output error falls at the same time, from 12.76% to 7.40%, and that is not SmoothQuant working. The output error weights every channel by how loud its activations are, and this scale raises exactly the loud channels, so it collects AWQ's benefit by accident while doing SmoothQuant's damage on purpose. With 16-bit activations that is a bad trade dressed as a good one, because there was no activation grid to rescue. Put the outlier magnitude back to 20x, untick SmoothQuant, and go and see it paid for instead: weight format INT8, granularity per-output-channel, activation format INT8, one scale per token. The layer error is 2.97%. Tick SmoothQuant and it is 0.877%, bought with weight tensor error rising from 0.781% to 1.26%. That is the whole method: a small, real cost on a side that had room, paid to avoid a large one on a side that did not.

AWQ's real finding is not the scaling. It is that which channels are salient can only be read off the activations. Their ablation keeps 1% of weights in FP16 and measures OPT-6.7B at INT3 group 128: chosen by activation magnitude, perplexity goes from 23.54 to 11.39; chosen by weight magnitude, 22.37; chosen at random, 24.23. The weight tensor does not know which of its columns matter. Only a forward pass over real data does, which is why every method here needs a calibration set.

GPTQ is not a scaling method at all. It fixes the scales and improves the rounding. Rounding each weight to its nearest level is optimal for that weight and not for the layer, because the errors are correlated through the inputs. GPTQ walks the input dimension one column at a time, and after quantizing a column it subtracts the error it just committed from all the columns it has not reached yet, weighted by the inverse Hessian (2XXᵀ)⁻¹ of the calibration activations. Individual weights end up further from their originals. The layer's output ends up closer. In the first simulation — restored to INT4, group 128, 16-bit activations, 20x outliers, SmoothQuant off — the layer error goes from 12.76% to 4.68% with the GPTQ toggle on, while the weight tensor error goes the other way, from 11.60% to 24.45%. That gap is the method, not a side effect of it.

Do not read that as a ranking. The number the simulation reports is layer reconstruction error, which is precisely the objective GPTQ minimises, so it is marking its own homework. On end-to-end perplexity the published ordering is different and much tighter. Llama-2-7B, WikiText, INT4 group 128: FP16 5.47, round-to-nearest 5.73, GPTQ 5.69, GPTQ with activation reordering 5.63, AWQ 5.60. Every one of those is a fine model. Turn AWQ on as well, leaving GPTQ on, and the layer error drops again, to 2.12% — which is the actual answer to "GPTQ or AWQ": they compose, because one picks scales and the other picks roundings.

LLM.int8() is the fourth approach and the one nobody uses in production, for an instructive reason. It does not transform anything. It finds the outlier dimensions at runtime — any dimension holding a value above a threshold the paper fixes at 6.0 — pulls them out into a separate 16-bit matmul, and runs the other 99.9% in 8-bit. It is exact where it matters and it needs no calibration. It is also two matmuls, one of them with a data-dependent shape, and the paper is candid that the overhead "can slow inference for models with less than 6.7B parameters". It answers the question correctly and pays for the answer at every forward pass, which is the trade SmoothQuant exists to avoid.

Where quality actually goes

The published numbers are more useful than any intuition about bits, because the shape of the degradation is not linear and not model-independent.

At INT4 group 128, the cost is small and roughly constant in relative terms. Llama-2-7B goes 5.47 → 5.73 perplexity with plain round-to-nearest, 5.60 with AWQ. Llama-2-70B goes 3.32 → 3.46 → 3.41. You can ship this.

At INT3 group 128 the same models go 5.47 → 6.66 and 3.32 → 3.98. Still functional, clearly worse. And then the floor gives way: OPT-1.3B at INT3 group 128 goes from 14.62 to 119.47, and OPT-2.7B from 12.47 to 298.00, while OPT-6.7B in the same table goes only to 23.54 and OPT-30B to 18.80. The collapse is not monotone in model size and it is not predictable from the bit width. Below four bits you are in a regime where you must measure your specific model, and the measurement is cheap compared to discovering it in production.

The KV cache is a separate quality decision with a separate answer. FP8 E4M3 on the cache is close to free because keys and values are activations, and activations have exactly the wide-dynamic-range, outlier-prone shape that an exponent handles and an even integer grid does not. This is why vLLM shipped FP8 cache quantization years before INT8 cache quantization and why the INT8 request is still open. The same argument applies to activations in the matmul. Untick GPTQ and AWQ first — the paragraphs above left both on, and AWQ divides the activations by its scale, which flattens exactly the outliers this comparison is about; with it on, INT8 activations read 0.718% against FP8's 1.69% and the result reverses. Then set the weight format to INT8 and the granularity to per-output-channel, so the weights contribute almost nothing and what you are reading is the activation format on its own. At 20x outliers, FP8 activations cost 1.92% and INT8 costs 2.97%. Push the outliers to 100x and FP8 improves, to 1.40%, while INT8 gets worse, to 3.26% — and the median ordinary channel is down to 20 of its 255 available levels. That levels readout is the honest one here: this layer's output is mostly made of the outlier channels, which an INT8 scale set by them represents perfectly well, so the 3.26% badly understates what has happened to the other 253. Leave the weights at INT4 group 128 instead and both formats read about 13%, because there the weights dominate and the activation format is not what you are measuring. FP8 spends three of its eight bits on a mantissa and four on an exponent, so its resolution is relative; INT8 spends all eight on evenly spaced steps, so its resolution is absolute and the outlier sets it for everyone.

Which is also why FP8 is not simply the better format. Put the activations back to 16-bit and the outlier magnitude back to 20x, then run the granularity ladder with FP8 weights: 2.68%, 2.65%, 2.69%, 2.75%, 2.37% from per-tensor to group 32. It hardly moves, because the exponent was already absorbing the dynamic range that grouping exists to remove. Then set the weight format to INT8 and watch per-output-channel INT8 (0.846%) beat per-output-channel FP8 (2.65%). Given a group whose values are all the same order of magnitude — which is what a per-channel or group-wise weight scale guarantees — evenly spaced levels are the better use of 256 codes. FP8's advantage is robustness to what you did not group away.

Checking it on a real system

Read the checkpoint before you read a benchmark. Open config.json and look at quantization_config: it tells you the method, the bit width, the group size, whether there is a zero point, and — in the modules_to_not_convert or ignore list — which layers were left alone. Then compute the effective bits yourself, bits + (16 + zp_bits) / group_size, and multiply by the linear-layer parameter count. If your prediction misses the file size by more than a header, something is not what the config says it is.

Then check what the engine chose to run, which is not always what you asked for. vLLM logs the resolved method and kernel at startup, and the difference between gptq_marlin and plain gptq is the entire distance between the two kernel-quality settings in the second simulation. If you passed a GPTQ checkpoint and the log does not say Marlin, you are on the slow path and your large-batch throughput will be below BF16.

Under load, the diagnosis is a two-step comparison, and both steps need the batch size printed next to them. Sample DCGM's DCGM_FI_PROF_DRAM_ACTIVE and DCGM_FI_PROF_PIPE_TENSOR_ACTIVE during steady-state generation. High DRAM activity with tensor-pipe activity in the low single digits means you are in the regime where weight-only quantization pays, and its speedup should be close to your byte ratio; if it is not, the kernel is the problem. Tensor-pipe activity above about 30% means you are compute-bound, and no amount of shrinking the weights will help — you need the activations quantized too, or fewer tokens per forward pass.

Finally, separate the two things people conflate when they say "it OOMs without quantization". Weight quantization changes a fixed cost; cache quantization changes a per-sequence cost. vLLM prints both consequences in one line at boot: GPU KV cache size: N tokens, followed by the maximum concurrency that implies. Quantize the weights and that number goes up because the pool grew. Quantize the cache and it goes up because each sequence got cheaper. At 4k contexts on an 8B model the second is worth twice the first, and they are set by different flags — --quantization and --kv-cache-dtype. Setting one and expecting the other is the most common version of this whole misconception.

Next: the allocator that decides how much of that cache you are actually using rather than reserving, paged attention; the scheduler that sets the batch size every number here depends on, continuous batching; and the other way to spend decode's idle tensor cores, speculative decoding. If the cache is what is hurting you, the architectural version of the same fix is grouped-query attention, and the way to split both weights and cache across cards is tensor parallelism. The forward pass whose token count decides all of this is sized by chunked prefill.