LLM Quantization

A large language model is billions of numbers. Store each one more coarsely, and the model gets several times smaller and faster.

Try it

Chapters · 14

01What it does now
02Weights fill the memory
03How many bits a number gets
04Rounding to a grid
05A ruler for each group
06A few very large numbers
07Correcting as it rounds
08Only −1, 0 and +1
09The KV cache, too
10What is lost
11What is gained
12Where it runs
13Taken apart
14Bench

Scroll

01

What it does now

OpenAI’s open-weight gpt-oss-120b, released in August 2025, has 116.8 billion parameters. More than 90 % of them, the experts’ weights, are stored in MXFP4 at 4.25 bits each; the whole model is 60.8 GiB and fits on one 80 GB GPU. Stored at 16 bits, the same parameters would take 234 GB.

At 16 bits, Llama 3.1 405B does not fit in one machine with 8 H100s; it takes two machines and 16 GPUs, and Meta also published a way to run it in FP8. Gemma 3 27B, in its quantization-aware-trained version, goes from 54 GB of weights to 14.1 GB and fits on one 24 GB RTX 3090. BitNet b1.58 2B4T was trained from scratch with every weight −1, 0 or +1; its weights outside the embeddings take 0.4 GB.

On the bench is a real block of weights: 24 × 48 numbers of the attention output matrix (3,584 × 3,584) in layer 14 of Qwen2.5-7B, read as published, in BF16. Each steel pin is one weight; its height is the weight’s size, bright for positive, dark for negative.

gpt-oss-120b: MXFP4, 4.25 bits, 60.8 GiB · Llama 3.1 405B: 16 GPUs at 16 bits · Gemma 3 27B: 54 → 14.1 GB · BitNet b1.58: −1 / 0 / +1

02

Weights fill the memory

A model’s size is, to a first approximation, its number of parameters times the bytes each one takes. Qwen2.5-7B has 7,615,616,512 parameters; at 16 bits, 2 bytes each, that is 15.2 GB. Llama 3.1 70B takes 141 GB and 405B takes 812 GB. One H100 has 80 GB of memory.

The gauge on the right keeps this account: one steel plate for each bit of a weight, 16 plates for 16 bits. The dial indicator reads the stack’s height; its inner ring converts it to GB per billion weights, exactly 2 GB at 16 bits.

To write each token, the model reads all of its weights from memory once. Half the bits per weight is half the space, and also half the data to move at every step.

Memory = parameters × bytes each · Qwen2.5-7B: 15.2 GB (16 bits) · one H100: 80 GB

03

How many bits a number gets

Neural networks are usually trained with 32-bit floating point: 1 sign bit, 8 exponent bits, 23 mantissa bits. BF16 keeps the 8 exponent bits and only 7 of mantissa, so it has the same range, more coarsely; FP16 has 5 exponent bits and 10 of mantissa and tops out at 65,504. Qwen2.5-7B and Llama 3.1 are both published in BF16.

Below that, FP8 comes in two kinds: E4M3 (4 exponent bits, 3 of mantissa, up to 448) and E5M2 (up to 57,344), the first recommended for weights and activations, the second for gradients. DeepSeek-V3 ran most of its heavy training arithmetic in FP8, all of it in E4M3. The integer formats INT8 and INT4 have no exponent: they are an evenly spaced ruler and need a separate scale.

A 4-bit float, FP4 (E2M1), can only be 0, 0.5, 1, 1.5, 2, 3, 4 or 6, or their negatives. MXFP4 lets 32 of them share one 8-bit power-of-two scale, 4.25 bits each on average; NVIDIA’s NVFP4 shares one FP8 scale among 16, about 3.5 times smaller than FP16.

FP32 1-8-23 · BF16 1-8-7 · FP16 1-5-10 · FP8 E4M3 / E5M2 · FP4 E2M1: 0, 0.5, 1, 1.5, 2, 3, 4, 6

04

Rounding to a grid

Storing a number as a 4-bit integer puts it on a ruler with only 15 marks, −7 to +7. A scale sets how wide one step is. The usual choice puts the largest absolute value exactly on ±7; every other number is divided by the step, rounded, and kept as an integer, then multiplied back when used.

Here all 1,152 weights on the tray share one ruler. The red pin is the largest of them, 0.118, and it sets the grid: one step is 0.118 ÷ 7 ≈ 0.017. The glass behind is ruled with the 8 heights a pin can now take, 0 to 7 steps; the colour keeps the sign.

The cost is error, up to half a step per number. Half the weights in this block are smaller than 0.015, less than one step, so 31 % of them round to 0 and the block’s relative error is about 21 %. A few large numbers stretch the ruler, and the small ones lose their resolution.

q = round(w ÷ s), s = largest |w| ÷ 7 · 4-bit integers: −7 … +7 · on the tray: 31 % of the weights become 0

05

A ruler for each group

The fix is more rulers. One per row (an output channel) is per-channel; one for every g numbers along a row is per-group. The AWQ paper uses groups of 128, and the GPTQ paper tried 128 and 1,024; llama.cpp’s Q4_0 puts 32 numbers in a group with one 16-bit scale.

The steel ribs cut each row into 3 groups of 16, and each brass tab stands as tall as its group’s tallest pin: that is the group’s ruler. At the same 4 bits, one ruler for the block gives a relative error of about 21 %, one per row 11 %, one per 16 weights 8 %.

Rulers take space too. One 16-bit scale per 16 numbers adds a whole bit per weight; one per 128 adds only 0.125. A study of more than 35,000 runs found that, for a fixed number of total bits, only two things reliably helped: splitting the parameters into small, separately quantized blocks, and the choice of number format.

One ruler per block / row / 16 weights: 21 % → 11 % → 8 % (on the tray, 4 bits) · a ruler costs 16 ÷ g bits

06

A few very large numbers

Weights are mostly tame; the trouble is the layer’s inputs, the activations. LLM.int8() (2022) found dimensions in the inputs with numbers up to 20 times larger than elsewhere; from about 6.7 billion parameters they appear in every layer, in about 6 fixed dimensions, roughly 0.1 % of all features. Zeroing those dimensions made perplexity 600 to 1,000 % worse.

LLM.int8() pulls those dimensions out and multiplies them in 16 bits, the other 99.9 % in 8, with no loss up to 175 billion parameters and half the memory. SmoothQuant goes another way: since the large values sit in fixed channels, divide each channel of the input by s and multiply that column of weights by s, which leaves the product unchanged; s = max input^α ÷ max weight^(1 − α), with α = 0.5 by default. Then weights and inputs both fit 8 bits, up to 1.56 times faster with half the memory.

The brass pins along the tray’s front are the layer’s 48 inputs, two of them very large; these inputs are made up by the page, the weights are real. Smoothed, those two inputs drop and their two columns of steel pins rise: the difficulty moves from the inputs to the weights. At 8 bits, the ordinary inputs used only 41 of the 255 levels; smoothed, they use 109.

LLM.int8(): about 0.1 % of dimensions in 16 bits · SmoothQuant: x ÷ s, w × s, α = 0.5 · the inputs here are made up

07

Correcting as it rounds

The simplest way is to round every number on its own, called RTN. GPTQ rounds one column at a time, and each time it spreads that column’s error over the columns not yet rounded, weighted by how the inputs move together, so that the layer’s output changes as little as possible. It uses second-order statistics of the inputs (the inverse of a Hessian), calibrated on just 128 pieces of text of 2,048 tokens each.

On one GPU it brings a 175-billion-parameter model down to 3 or 4 bits in about four hours. OPT-175B’s perplexity (lower is better) is 8.34; at 4 bits RTN gives 10.54 and GPTQ 8.37; at 3 bits RTN collapses to thousands, GPTQ gives 8.68.

The bridge sweeps from left to right; as it passes a column, that column snaps to the grid and the pins to its right shift a little (the inputs here are made up). At 4 bits in groups of 16, the error of the weights themselves rises from 8 % to 14 %, while the error of the layer’s output falls from 8 % to 5 %. AWQ takes another path: it found that the 1 % of weight channels picked by the inputs’ sizes matter most, and scales each channel’s weights up and its input down by the input’s mean size before rounding.

GPTQ · OPT-175B: 16 bits 8.34, 4 bits RTN 10.54, GPTQ 8.37 · on the tray: output error 8 % → 5 %

08

Only −1, 0 and +1

Everything so far quantizes a model after training. The other road trains the model for fewer bits, called quantization-aware training. Gemma 3’s QAT versions trained for about 5,000 more steps and cut the perplexity lost at 4 bits by 54 %.

BitNet b1.58 goes furthest: trained from scratch, every weight can only be −1, 0 or +1. The matrix is divided by γ, the mean absolute value of its weights, rounded, and clipped to −1 … 1. Three values carry log₂3 ≈ 1.58 bits. In the paper, a 3-billion-parameter model of this kind reached a perplexity of 9.91 against 10.04 for a 16-bit model trained on the same data, with 3.55 times less memory and 2.71 times faster per token.

The pins are rounded the same way here: γ = 0.0177, every pin takes one height, about a third drop to the deck, and the gauge keeps 1.58 plates. But this forces already-trained weights to three values, with an error of about 56 %; BitNet is trained under that constraint. In storage, four ternary values go into one byte, 2 bits each.

W̃ = RoundClip(W ÷ γ, −1, 1) · log₂3 ≈ 1.58 bits · BitNet b1.58 3B: memory ÷ 3.55, 2.71 times faster per token

09

The KV cache, too

While generating, a model also keeps the keys and values of every earlier token in every layer: the KV cache. Each token stores two of them per layer, each the number of KV heads × the head size. Qwen2.5-7B has 28 layers, 4 KV heads and 128 numbers a head, so at 16 bits each token takes 56 KiB; 131,072 tokens take 7.5 GB, half as much as the weights.

The longer the context, the larger the cache’s share. KIVI found that keys should be quantized per channel and values per token, and reached 2 bits with almost no loss: 2.6 times less peak memory, batches up to 4 times larger, 2.35 to 3.47 times the throughput. KVQuant used 3 bits, lost less than 0.1 in perplexity, and let LLaMA-7B handle a million tokens on one 80 GB A100.

Fewer KV heads also save cache: Llama 3’s 8B model gives its 32 attention heads only 8 KV heads, so 131,072 tokens take 17 GB; with a key and value for every head they would take 69 GB.

Per token: 2 × layers × KV heads × head size × bytes · Qwen2.5-7B, 131,072 tokens: 7.5 GB (16 bits) → 0.94 GB (2 bits)

10

What is lost

Perplexity measures how surprised a model is by real text; lower is better. On LLaMA3-8B (WikiText2, 6.1 at 16 bits), 4 bits in groups of 128 gives RTN 8.5, GPTQ 6.5 and AWQ 6.6. At 3 bits: RTN 27.9, GPTQ and AWQ both 8.2. At 2 bits they all collapse: GPTQ 210, AWQ about 1.7 million.

A study of more than 35,000 runs, from 19 million to 176 billion parameters, found that for the same total number of bits, 4 bits per parameter was almost always best; at 3 bits the trend reversed. Another study, on models up to 1.7 billion parameters, found that the more data a model was pretrained on, the more it lost when quantized afterwards.

Close averages do not mean the same behaviour. One study found that with accuracy within 2 %, 5 % or more of the answers can still flip from right to wrong or wrong to right; Meta wrote that standard benchmarks did not reflect FP8’s effects well, and compared reward-model scores of 100,000 responses instead.

LLaMA3-8B (16 bits: 6.1) · 4 bits: RTN 8.5 / GPTQ 6.5 / AWQ 6.6 · 3 bits: RTN 27.9 · 2 bits: collapse

11

What is gained

Memory first. Gemma 3 27B goes from 54 GB to 14.1 GB; gpt-oss-120b from 234 GB at 16 bits to 65 GB; Llama 3.1 405B in FP8 is about 487 GB, less than the 640 GB in one machine with 8 H100s.

Then speed. Writing one token at a time, the chip’s arithmetic sits mostly idle; the bottleneck is moving the weights in from memory. An RTX 4090 moves 1 TB a second, and this step does about one operation per byte moved. So tokens a second have a ceiling: bandwidth ÷ bytes of weights. Qwen2.5-7B on an M5 Max at 614 GB/s tops out near 40 tokens a second at 16 bits, near 143 at 4 bits.

Measured speeds sit below the ceiling but move the same way: with GPTQ, 3-bit OPT-175B takes 71 ms a token on one A100, against 230 ms on five A100s at 16 bits; AWQ’s inference engine is 3.2 to 3.3 times faster on average on desktop and mobile GPUs.

Tokens a second ≤ bandwidth ÷ bytes of weights · Qwen2.5-7B at 614 GB/s: about 40 (16 bits) → about 143 (4 bits)

12

Where it runs

To run a model on your own computer, the usual tools are llama.cpp and its GGUF files: one file with the weights and all their metadata, mapped straight into memory. It offers a long list of quantized types: Q8_0 groups 32 numbers with one 16-bit scale, 8.5 bits each; Q4_K_M mixes block types across tensors, 4.89 bits a weight and 4.58 GiB for Llama 3.1 8B, against 14.96 GiB at 16 bits.

Apple’s MLX quantizes to 4 bits in groups of 64 by default and also offers MXFP4 and NVFP4. BitNet has its own CPU library, 1.37 to 5.07 times faster than 16 bits on ARM. Quantized, a 13-billion-parameter model runs on a laptop GPU with 8 GB of memory, and Llama-2-70B on a 64 GB Jetson Orin.

The speeds on this page are each team’s own measurements, on different hardware and settings, and do not compare across sources; what compares is ratios within one table: on the same machine, Q4_K_M is 3.3 times smaller than 16 bits and 2.5 times faster at writing.

GGUF · Q8_0: 8.5 bits · Q4_K_M: 4.89 bits (Llama 3.1 8B) · MLX: groups of 64 · MXFP4 / NVFP4

13

Taken apart

The bench comes apart in eight pieces: the pin tray (the weights, with its group ribs, brass scale tabs and the input pins along its front), the ruled glass (the grid), the bridge and its two rails (rounding column by column), the plate gauge, the dial indicator on its stand, the bits selector, and the nameplate on the granite’s front.

The pins are 5.7 mm across on a 6.8 mm pitch; 1 mm of height stands for a weight of 0.00116. The 1,152 pins are 24 neighbouring rows and 48 columns of one matrix, rows marked on the left, column numbers (416 to 463) along the front.

A gauge plate is 3.6 mm; 16 of them are 16 bits. The dial turns through 300° from 0 to 16 bits.

Pin tray · ruled glass · bridge · rails · plate gauge · dial indicator · bits selector · nameplate

14

Bench

The bench quantizes the whole block of 32 × 512 = 16,384 real weights; the tray shows 24 × 48 of them. At the top right is the distribution of all of them, with the first group’s grid in red.

Bits run from 16 down to 2, or 1.58 (ternary); a group can be 16, 32, 64 or 128 weights, a whole row (512), or one ruler for the block. The readouts are the relative error, the share rounded to 0, the bits per weight counting the scales, and how large Qwen2.5-7B would be at that many bits.

Scales count as 16 bits; the grid is symmetric and set by each group’s largest absolute value; ternary uses each group’s mean absolute value. This is the simplest rounding, not GPTQ or AWQ.

The whole block of 16,384 weights · readouts: error, share of zeros, bits per weight, model size

Try thisSet 3 bits, then widen the group from 16 weights to the whole block. How much does the error grow, and how many more weights round to 0?

What the model leaves out

The tray holds 1,152 numbers of one matrix (the bench 16,384), where Qwen2.5-7B has 7.6 billion; a pin’s height is a weight’s absolute value, and colour gives the sign. “One ruler for the block” means this block; real per-tensor quantization covers the whole 3,584 × 3,584 matrix. The inputs on the front wall and the calibration inputs GPTQ and SmoothQuant use here are random numbers made up by the page (with two very large channels); GPTQ is computed column by column as the paper’s algorithm, without its batched updates for speed. The bridge is a picture, though GPTQ does round one whole column at each step. In the smoothing chapter all the pins are drawn a little shorter, so that the two columns that rise still fit. The bench only does the simplest rounding (symmetric, set by the largest value); the floating-point formats are only described in the text.

Easy to get wrong

The very large numbers are in the activations (the inputs), not the weights; AWQ’s “important weights” are picked by the sizes of the inputs. 1.58 bits is the information in one ternary value; storage usually takes 2 bits each. Q4_K_M is not 4.5 bits a weight; for Llama 3.1 8B it averages 4.89. Bandwidth ÷ bytes is a ceiling on tokens a second, before the KV cache and other costs. Perplexities from different papers use different data and settings and do not compare across tables, only within one. Quantization-aware training and quantizing after training are different things: BitNet’s results come from training from scratch, and do not show that any model can be squeezed to 1.58 bits afterwards.

Common questions · 4

How much memory do a language model’s weights take?: A model’s size is, to a first approximation, its number of parameters times the bytes each one takes. Qwen2.5-7B has 7,615,616,512 parameters; at 16 bits, 2 bytes each, that is 15.2 GB.
02Weights fill the memory
How is a weight quantized to 4 bits?: Storing a number as a 4-bit integer puts it on a ruler with only 15 marks, −7 to +7. A scale sets how wide one step is.
04Rounding to a grid
What does the 1.58 in BitNet b1.58 mean?: BitNet b1.58 goes furthest: trained from scratch, every weight can only be −1, 0 or +1. The matrix is divided by γ, the mean absolute value of its weights, rounded, and clipped to −1 … 1. Three values carry log₂3 ≈ 1.58 bits.
08Only −1, 0 and +1
How much does a language model lose at 4 bits?: On LLaMA3-8B (WikiText2, 6.1 at 16 bits), 4 bits in groups of 128 gives RTN 8.5, GPTQ 6.5 and AWQ 6.6.
10What is lost

Sources

Data

The weights on the tray are real: a 24 × 48 window (the bench: 32 × 512) of one weight matrix of Qwen2.5-7B (Apache 2.0; layer 14, attention output projection), read as the published BF16 bits (docs/plates/quant/weights.py). The rounding, the errors and GPTQ’s sweep are computed by the page (src/topics/quant/science.js). The layer’s inputs on the tray’s front wall, with their two large channels, and the calibration inputs GPTQ and SmoothQuant use here are made up by the page. Material presets, parts and the softbox studio of the site’s model kit are ported from AirsupHQ/machine-skill (MIT, © 2026 Air Supply Technologies Inc.).

The text, pictures and models on this page are licensed CC BY-NC 4.0: free to use and adapt for teaching and other non-commercial purposes, with credit to world.lapla.si. Third-party material listed above keeps its own licence.