# Tokens and Scaling Laws

Laplace’s Mini World · Plate F26 · AI

> A language model reads tokens, not letters. Up front, byte-pair encoding runs in your browser; behind it stand 34 small models trained for this page.

- Interactive model: https://world.lapla.si/en/scaling/
- Embed: https://world.lapla.si/embed/en/scaling/
- 中文: https://world.lapla.si/zh/scaling/
- Updated: 2026-10-20

## 01 Models read tokens

A language model takes in a string of numbers. Each number stands for a token: a common word, part of a word, a punctuation mark, or a few bytes. GPT-4o’s tokenizer has 200,019 of them; “All human beings are born free and equal in dignity and rights.” is 13 tokens to it.

Each ceramic tile on the rail is one token. The space before a word belongs to the word, so “ human” and “human” are two different numbers.

*GPT-4o’s tokenizer (o200k\_base) · 63 characters → 13 tokens*

## 02 The most frequent pair

Byte-pair encoding starts from single bytes: one per letter, three per Chinese character. Count every pair of neighbouring pieces in the training text, join the most frequent pair into a new piece and write the rule down; count again, join again, tens of thousands of times.

This tokenizer was learned a moment ago in your browser, from the first two chapters of Alice’s Adventures in Wonderland. Its first rule is space + t (614 times), its second h + e (559), and its eighth joins “ t” and “he” into “ the”. Each press of the gantry is one rule applied.

*Count neighbouring pairs → join the most frequent → write the rule → repeat*

## 03 More rules, shorter lines

The two chapters are 22,182 characters, 22,791 bytes. After 100 rules they take 12,044 tokens; after 250, 9,340; after 500, 7,672, about 2.9 characters a token.

More rules make a bigger vocabulary and shorter lines, so a model fits more text in at once; but every number needs its own row of weights, and rare numbers are learned from few examples. GPT-2’s vocabulary has 50,257 numbers, GPT-4’s 100,277, GPT-4o’s 200,019.

*This page’s tokenizer: 0 rules 22,791 · 100 rules 12,044 · 250 rules 9,340 · 500 rules 7,672 tokens*

## 04 How many r’s in strawberry

Give GPT-4o’s tokenizer “How many r’s are in strawberry?” and it makes 8 tokens; “ strawberry”, with the space in front, is a single number. That number is what the model gets, not the ten letters s, t, r, a, w…

At the start of a line, with no space, the same word is cut into st, raw and berry; GPT-4’s tokenizer cuts it into str, aw and berry. To count letters, a model has to have learned how each number is spelled. This is often given as the reason models miscount letters; it is one reason among others.

*“ strawberry” → 1 token · “strawberry” → st | raw | berry (GPT-2, GPT-4o)*

## 05 One sentence, three prices

Article 1 of the Universal Declaration of Human Rights is 170 characters in English and 33 tokens to the GPT-2, GPT-4 and GPT-4o tokenizers alike. The Chinese text is 43 characters: 86 tokens for GPT-2, 50 for GPT-4, 35 for GPT-4o. The Japanese is 84 characters: 116, 92 and 71.

GPT-2’s tokenizer learned mostly from English web pages; many Chinese characters have no number of their own and fall apart into two or three pieces of their UTF-8 bytes. This page’s tokenizer, which has read only Alice, is the extreme case: no Chinese merges at all, three tokens a character. The token count sets how much a model can read, how much it computes, and what it costs when it is billed by the token.

*Article 1 · English 33 / 33 / 33 · Chinese 86 / 50 / 35 · Japanese 116 / 92 / 71 tokens (GPT-2 / GPT-4 / GPT-4o)*

## 06 Error falls as a power law

In 2020, Kaplan and colleagues at OpenAI trained language models from 768 to 1.5 billion parameters (not counting the vocabulary). The test error fell along a power law in the parameters N, the data D and the compute C, each on its own: L(N) = (8.8 × 10¹³ / N)^0.076 and L(D) = (5.4 × 10¹³ / D)^0.095. Ten times the parameters multiplies the error by 0.84.

The row of engines behind is this page’s own experiment: 11 sizes, from 20 thousand to 10 million parameters. The best engine at each budget goes from 4.27 to 3.26 (nats per token; lower is better). Each tenfold step in compute multiplies the best error by 0.87. This is a small range; Kaplan’s paper also says such trends must eventually level off, because language has an irreducible uncertainty of its own.

*Best error per budget · 10¹² 4.27 · 3 × 10¹² 3.99 · 10¹³ 3.70 · 3 × 10¹³ 3.46 · 10¹⁴ 3.26 nats*

## 07 Compute ≈ 6ND

The compute it takes to train a model is close to C ≈ 6ND: per parameter and per token, 2 floating-point operations forward and 4 backward (see the plate How a Neural Network Learns). The lever at the back left picks a budget, five steps from 10¹² to 10¹⁴ operations.

At one budget, twice the parameters means half the tokens. Chinchilla, 70 billion parameters reading 1.4 trillion tokens, took about 5.9 × 10²³; this page’s largest budget, 10¹⁴, is about one six-billionth of that.

*C ≈ 6 · N · D · 2 forward + 4 backward per parameter per token*

## 08 The same compute, shared out

The lever is at 3 × 10¹³. Each hopper holds the tokens its engine can read at that budget: the smallest reads 30 million, the largest only 490 thousand. They start together and finish together; the ink in each tube is its error on four books it never saw.

Too small, and an engine cannot hold what it reads; too big, and it has not read enough. The lowest is in the middle: 430 thousand parameters, 11.5 million tokens, 3.46 nats. The red thread is a parabola through the tube tops near it; its lowest point is at about 410 thousand parameters. This is the second of the methods Hoffmann and colleagues used in 2022 to size Chinchilla.

*isoFLOP profile · 3 × 10¹³ operations · lowest at 430k parameters / 11.5M tokens*

## 09 Twenty to one

At DeepMind, Hoffmann and colleagues trained over 400 models, from 70 million to over 16 billion parameters, on 5 to 500 billion tokens. All three of their methods said the same: with more compute, grow parameters and tokens in about equal proportion; twice the parameters, twice the data. In their Table 3, 1 billion parameters go with 20.2 billion tokens and 10 billion with 205.1 billion: about 20 tokens a parameter.

With Gopher’s compute they trained Chinchilla, 70 billion parameters on 1.4 trillion tokens, against Gopher’s 280 billion on 300 billion. Chinchilla did better on a wide range of tasks, 67.5% on average on MMLU, over 7 points above Gopher, and it is cheaper to run. This page’s small models come out at 25 to 80 tokens a parameter at their best, fewer as compute grows; models this small should not be held against the 20.

*Gopher 280B parameters × 300B tokens · Chinchilla 70B × 1.4T · the same compute*

## 10 One engine, cut open

Cut open, the largest engine is 8 rings, one Transformer layer each, 320 wide. A layer has about 12d² weights: 4d² in attention (the thin brass disc) and 8d² in the feed-forward network (the thick copper one). How attention and the feed-forward block compute is on the plate Transformer Attention.

There is also a table of 1,024 rows: looked up on the way in, and used again on the way out to score every number. In small models this table is most of the parameters; the smallest engine has 16 thousand of its 20 thousand there. Kaplan counted only the parameters in the layers, Hoffmann counted them all, and this was later found to be one of the main reasons their answers differ.

*A layer ≈ 12d² · attention 4d² · feed-forward 8d² · vocabulary 1024 × d*

## 11 Where the two laws part

Kaplan found that with more compute, parameters should grow faster than data: N\_opt ∝ C^0.73. Hoffmann found they should grow equally: ∝ C^0.50. By the first, GPT-3’s generation of models was a little large; by the second, it was badly undertrained.

Re-examinations in 2024 found causes: Kaplan counted only non-embedding parameters, at small scale (Pearce and Song); the last layer’s compute, the learning-rate warm-up and optimizer settings tuned with scale (Porian and colleagues). Epoch AI refitted Chinchilla’s data and found the paper’s third method inconsistent with its first two; corrected, it too gives about 20 tokens a parameter. This page’s family gives N\_opt ∝ C^0.63, closer to Kaplan: at this small scale, where the vocabulary table is a large share, the answer leans his way.

*N\_opt ∝ C^a · Kaplan 0.73 · Chinchilla 0.50 / 0.49 / 0.46 · this page 0.63*

## 12 Will the data run out?

A power law says what more data would buy, not that more data exists. Villalobos and colleagues estimate the effective stock of public human-written text at about 3 × 10¹⁴ tokens (90% range 1 to 10 × 10¹⁴); at current trends, training sets reach it some time between 2026 and 2032.

Short of data, a model can read it again. Muennighoff and colleagues found that four passes over the same data are nearly as good as four times the new data; by about sixteen passes, more compute adds almost nothing. On this page most models read only part of the data; the one that read most went through it 1.09 times.

*Effective stock ≈ 3 × 10¹⁴ tokens · 4 passes ≈ new data · returns near zero after about 16*

## 13 Compute at answer time

Compute can also be spent when answering: letting a model take more steps, or write several answers and pick one. Snell and colleagues compared this on maths problems in 2024: on problems the base model could already solve some of the time, extra compute at answer time beat a model with 14 times the parameters; on the hardest problems, more training was the better buy.

Training’s scaling laws measure error on a fixed test text. What answer-time compute buys changes a great deal with how hard the problem is, and there is no curve as steady for it yet.

*Compute at answer time · worth 14× the parameters on middling problems · the hardest still need training*

## 14 Taken apart

An engine is a foot, a stack of rings, a hopper on top and a work lamp; at the front, the token rail with its gantry and ram, then the tube bench and the compute lever. Each part can be lifted out on its own.

## 15 Laboratory

Pick a compute budget, change how many rules the tokenizer uses, or type a line and see how many pieces it becomes. The best size in the readings comes from this page’s measured isoFLOP profile; beside it is what the fitted formula predicts.

**Try this**: On the bench, pull the merges down to 0, then back to 500: how many fewer tokens does the same line take? Try a line in Chinese.

## What the model leaves out

The tokenizer has learned only 500 rules (real ones have tens of thousands to two hundred thousand), from two chapters of Alice, so outside English it hardly merges at all. Its pre-splitting simplifies GPT-2’s: white space is folded into single spaces, then text is cut into runs of letters, digits and other signs; contractions get no rule of their own. The model family is small: at most 10 million parameters, at most 38.4 million tokens, 64 tokens of context, each setting trained once. At one budget a big model reads fewer tokens and so takes fewer steps (as few as about 160), which makes the big side look worse. N counts the vocabulary table (shared by input and output) and leaves out the 64-row position table. Error is measured on four held-out novels in nats per token, and cannot be compared directly with numbers for other tokenizers. The engines’ sizes are a sketch: one ring per layer, wider for wider layers, not in proportion to the parameters; the hoppers fill on a log scale of the tokens (100 thousand empty, 40 million full).

## A common misunderstanding

A scaling law is not a law of nature. It is a curve measured for one kind of model, one kind of data and one way of training, and its exponents change when those do; the disagreement between Kaplan and Hoffmann came from how they worked. It describes how one number, the test error, falls; it does not say directly whether a model gains a particular skill. Twenty tokens a parameter is what Chinchilla’s setting gave; many models now read far more on purpose, so that a smaller model is cheaper to run.

## Common questions

### Do language models read letters or tokens?

A language model takes in a string of numbers.

See: [01 Models read tokens](https://world.lapla.si/en/scaling/#read)

### Why do models miscount the r’s in strawberry?

That number is what the model gets, not the ten letters s, t, r, a, w…

See: [04 How many r’s in strawberry](https://world.lapla.si/en/scaling/#straw)

### Does Chinese take more tokens than English?

The Chinese text is 43 characters: 86 tokens for GPT-2, 50 for GPT-4, 35 for GPT-4o.

See: [05 One sentence, three prices](https://world.lapla.si/en/scaling/#lang)

### Where does Chinchilla’s twenty tokens a parameter come from?

In their Table 3, 1 billion parameters go with 20.2 billion tokens and 10 billion with 205.1 billion: about 20 tokens a parameter.

See: [09 Twenty to one](https://world.lapla.si/en/scaling/#chinchilla)

## Sources

1. [Sennrich, Haddow and Birch (2016): Neural Machine Translation of Rare Words with Subword Units, arXiv:1508.07909 (merging the most frequent pair, vocabulary = start + merges, no merges across words)](https://arxiv.org/abs/1508.07909)
2. [Gage (1994): A New Algorithm for Data Compression, C Users Journal (byte-pair encoding)](https://dl.acm.org/doi/10.5555/177910.177914)
3. [Radford et al. (2019): Language Models are Unsupervised Multitask Learners (byte-level BPE, no merges across character classes, 50,257 tokens)](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)
4. [OpenAI: tiktoken (gpt2, cl100k\_base, o200k\_base; GPT-4 uses cl100k\_base, GPT-4o o200k\_base)](https://github.com/openai/tiktoken)
5. [Kaplan et al. (2020): Scaling Laws for Neural Language Models, arXiv:2001.08361 (eqs. 1.1–1.3, N\_opt ∝ C^0.73, C ≈ 6N per token, trends must level off)](https://arxiv.org/abs/2001.08361)
6. [Hoffmann et al. (2022): Training Compute-Optimal Large Language Models, arXiv:2203.15556 (over 400 models, three methods, Tables 2 and 3, eq. 10, Chinchilla and Gopher)](https://arxiv.org/abs/2203.15556)
7. [Besiroglu et al. (2024): Chinchilla Scaling: A replication attempt, arXiv:2404.10102 (refitted E, A, B, α, β; about 20 tokens a parameter)](https://arxiv.org/abs/2404.10102)
8. [Pearce and Song (2024): Reconciling Kaplan and Chinchilla Scaling Laws, arXiv:2406.12907 (non-embedding parameters and small scale)](https://arxiv.org/abs/2406.12907)
9. [Porian et al. (2024): Resolving Discrepancies in Compute-Optimal Scaling of Language Models, arXiv:2406.19146 (last-layer compute, warm-up, optimizer settings)](https://arxiv.org/abs/2406.19146)
10. [Villalobos et al. (2024): Will we run out of data? Limits of LLM scaling based on human-generated data, arXiv:2211.04325 (effective stock about 3 × 10¹⁴ tokens, 2026–2032)](https://arxiv.org/abs/2211.04325)
11. [Muennighoff et al. (2023): Scaling Data-Constrained Language Models, arXiv:2305.16264 (4 passes, about 16)](https://arxiv.org/abs/2305.16264)
12. [Snell et al. (2024): Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters, arXiv:2408.03314 (14×, training wins on the hardest problems)](https://arxiv.org/abs/2408.03314)
13. [United Nations: the Universal Declaration of Human Rights, English and Chinese texts](https://www.un.org/en/about-us/universal-declaration-of-human-rights)
14. [United Nations Information Centre, Tokyo: the Japanese translation](https://www.unic.or.jp/activities/humanrights/document/bill_of_rights/universal_declaration/)
15. [Project Gutenberg: Alice’s Adventures in Wonderland and the 105 public-domain English books used for training](https://www.gutenberg.org/)

## Data

The tokenizer on the rail is the site’s own byte-pair encoding (src/topics/scaling/bpe.js), learned in the browser from the first two chapters of Lewis Carroll’s Alice’s Adventures in Wonderland (Project Gutenberg, public domain). Byte-pair encoding: Gage (1994), Sennrich, Haddow and Birch (2016); byte-level, split by character class, as in GPT-2. The published tokenizers’ pieces and counts (gpt2, cl100k\_base, o200k\_base) were computed with OpenAI’s tiktoken (MIT licence) by src/topics/scaling/tokens.py. The model family is the site’s own: 34 GPT-shaped models trained with PyTorch by src/topics/scaling/train.py on 105 public-domain English books from Project Gutenberg, held out four of them for testing. The method follows Hoffmann et al. (2022): fixed compute budgets, isoFLOP profiles and a parametric fit. The sentence in three languages is Article 1 of the Universal Declaration of Human Rights: English and Chinese from un.org, Japanese as published by the United Nations Information Centre, Tokyo. Material presets, parts and the softbox studio of the site’s model kit are ported from AirsupHQ/machine-skill (MIT, © 2026 Air Supply Technologies Inc.).

## Licence CC BY-NC 4.0

The text, pictures and models on this page are licensed CC BY-NC 4.0: free to use and adapt for teaching and other non-commercial purposes, with credit to world.lapla.si. Third-party material listed above keeps its own licence.

https://creativecommons.org/licenses/by-nc/4.0/deed.en

## Cite this plate

Laplace’s Mini World, Plate F26, “Tokens and Scaling Laws”, https://world.lapla.si/en/scaling/, CC BY-NC 4.0, updated 2026-10-20.