Reasoning Models

Writing the Steps Down

In 2022, Wei and colleagues put eight worked examples with their steps into a prompt, and the model followed them: steps first, then the answer. They called the series of intermediate steps a chain of thought. On GSM8K, grade-school maths word problems, the 540-billion-parameter PaLM went from 17.9% answering directly to 56.9%, above the 55% of fine-tuned GPT-3 with a verifier, the best before. The same year, Kojima and colleagues found that adding “Let’s think step by step” alone helps too: text-davinci-002 went from 10.4% to 40.7% on GSM8K.

The gain only appeared in large models: among the models they tested, from about 100 billion parameters; smaller models wrote steps that were fluent but illogical. One advantage the authors list is that intermediate steps let problems that need more steps get more computation.

The task on the bench is to add up 3 to 8 single digits, and the checker accepts only the exact sum. The writer’s mental arithmetic is the bench’s assumption: the more numbers it adds on one line, the likelier a slip, and a sum written down is not forgotten. The four writers on the left write the answer at once; the four on the right write a line for each number they add. With seven numbers, 34% of the answers written at once are right, and 74% of those written line by line.

PaLM 540B · GSM8K 17.9% → 56.9% · seven numbers on the bench: 34% at once, 74% line by line

More Time to Think

In September 2024, OpenAI introduced o1 (releasing an early version, o1-preview, first), which “thinks before it answers”, writing a long internal chain of thought first. They write that its performance consistently improves with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute); both axes of their figure are log scale, with no values. On the 2024 AIME maths contest, GPT-4o solved 12% on average; o1 solved 74% with one sample a problem, 83% taking the consensus of 64, and 93% re-ranking 1,000 samples with a learned scoring function.

The same year, Snell and colleagues, using PaLM 2-S* on MATH problems, found that what test-time compute buys depends on how hard the problem is: spending it by difficulty was more than four times as efficient as drawing N answers and keeping the best. On problems a small model already solves at a fair rate, the same compute can beat a model 14 times its size. On the hardest problems more test-time compute barely helped, and more pretraining did more (train-time scaling is on the plate Tokens and Scaling Laws).

The brass bar on the bench caps the lines of thinking: when a strip reaches it, the writer must answer. The s1 paper (Muennighoff and colleagues, 2025) calls this budget forcing, and uses it the other way too: appending “Wait” to the thinking makes the model think on, and took AIME 2024 from 50% to 57%. On the bench, the trained writer gets 24% of eight-number sums right with no lines allowed and 88% with up to 10; four-number sums reach 94% at 4 lines and stay there.

o1 · AIME 2024: 74% one sample, 83% consensus of 64, 93% best of 1,000 · eight numbers on the bench: 0 lines 24%, 10 lines 88%

Checked by a Rule

How does a model learn to write steps that are long and useful? DeepSeek-R1 (posted in January 2025, published in Nature that September) uses reinforcement learning: the model writes, a rule scores it, and what scores well is reinforced. For maths, the final answer has to come in a set format (for example in a box), so it can be compared with the reference answer; for code, a compiler runs it against tests written beforehand. A format reward asks for the thinking to sit between <think> and </think>. Apart from that format, R1-Zero’s reward looks only at the final answer, with no constraint on the thinking.

On reasoning tasks they did not use a neural network to score (an outcome or a process reward model), because in large-scale reinforcement learning a reward model is open to being exploited by the policy, and retraining one costs substantial compute (reward models and their loopholes are on the plate Learning from Human Feedback). Tülu 3 (Lambert and colleagues, 2024) gave this kind of training a name: reinforcement learning with verifiable rewards (RLVR), where a response earns a reward only when it is verified to be correct.

The checker on the bench is a row of plungers at the foot of the strips, linked to the key box at the right end of the key desk. The box holds this question’s sum; the plungers read the answers one strip at a time and stamp a tick where they match and a cross where they don’t. It reads only the last line.

Right 1, wrong 0 · plus a format reward: thinking inside <think> · no reward model for reasoning tasks

A Group of Answers

R1’s reinforcement-learning algorithm is GRPO, which DeepSeek introduced in DeepSeekMath in 2024. The usual PPO trains a separate value model to estimate how much a question will score; it is typically as large as the policy, and costs memory and computation. GRPO does without it: it samples a group of answers to the same question and takes the group’s average as the baseline. R1 sampled 16 answers a question, 32 questions a step.

Each answer’s advantage is  = (r − mean) / standard deviation: right answers sit above the mean, with  positive; wrong ones below. With seven right out of eight, the mean is 0.875: the right ones are only a little above it (+0.38), the wrong one far below (−2.65). With one right out of eight it is the other way round. When all are right, or all wrong, every  is 0: that question teaches nothing.

A group on the bench is eight strips (G = 8). Each of the eight typewriter keys on the key desk rises or dips by its strip’s Â, and the dial at the left end points at the group’s mean.

 = (r − mean) / std · no value model · all right or all wrong: every  is 0

Above the Mean, More Often

The stops on the two wings are the policy. The left wing’s four rows and the right wing’s top three are each for how many numbers are still to add, and offer four moves: add one on a written line, add two, add three, or add the rest in the head and answer. How far a stop is drawn is the chance of that move, and a row’s draws add up to one. The right wing’s two lower rows decide whether to check a line just written, and whether to write one answer or several.

In an update, every decision on a strip gets that strip’s Â. Where  is positive, the stops it used are drawn out, so the move becomes more likely; where it is negative they are pushed in. Each push is averaged over the decisions on the strip and then over the group. As in PPO, the ratio of new to old policy is clipped to within 1 ± ε, and a KL penalty pulls the policy towards a reference model (both are on the plate Learning from Human Feedback). For R1-Zero the KL coefficient was 0.001, and every 400 steps the reference model was replaced by the latest policy.

Each push on the bench is one update. The untrained writer, wherever it has the choice, answers at once 64% of the time, and checks a line or hedges its answer only 5% of the time each.

Each decision × its strip’s  · clipping as in PPO · the KL penalty added to the loss

Straight from the Base Model

R1-Zero was trained like this: take the pretrained DeepSeek-V3-Base, skip supervised fine-tuning, and go straight to reinforcement learning. The first arXiv version (January 2025) reports AIME 2024 pass@1 rising from 15.6% to 71.0%; the Nature version trains for 10,400 steps (a step being one policy update) and reaches 77.9%, and 86.7% taking the majority of several answers.

During training the answers grew longer, from hundreds of tokens to thousands. The paper says the model learned by itself to spend more thinking time on harder problems; nothing in the reward asked for length. The Nature version’s methods also say the length cap was raised from 32,768 to 65,536 tokens at step 8,200, and both score and length jump there.

The bench trains its writer the same way from the start, about 90 updates a second, each one question and eight strips. After 1,500 updates, accuracy rises from 56% to 92%, and each strip’s thinking grows from 0.5 lines to 4.6, about 3 tokens to 32. Nothing in the bench’s reward is about length either.

R1-Zero · AIME 2024 15.6% → 71.0% (arXiv v1) / 77.9% (Nature) · bench 56% → 92%, 0.5 → 4.6 lines

“Wait”

The R1 paper shows a piece of model output: halfway through, it writes “Wait, wait. Wait. That’s an aha moment I can flag here.” and goes back over the steps. The authors call it an aha moment, from an intermediate version of R1-Zero in training, and say the model learned to rethink in an anthropomorphic tone. The Nature version counts the word “wait”: virtually absent early in training, sporadic between steps 4,000 and 7,000, and far more frequent after step 8,000.

Others found something different. Liu and colleagues (2025) saw DeepSeek-V3-Base, before any reinforcement learning, already writing “Aha” and “wait”; R1-Zero reflects more often, but its reflection is not positively correlated with higher accuracy.

On the bench, checks start at 5%. Between about 650 and 750 updates, check lines per strip jump from 0.6 to 1.5, and end at about 2.1. The bench’s check is a rule written in: it redoes the line just written and, if that line was wrong, fixes it nine times in ten, so on the bench it pays. How much reflection helps real models, the two studies above disagree.

“wait”: virtually absent before step 4,000, far more after 8,000 · check lines per strip on the bench 0.02 → 2.1

Examples First

R1-Zero’s answers were hard to read, and sometimes mixed English and Chinese in one chain of thought. DeepSeek-R1 changed the recipe: first supervised fine-tuning on thousands of long chain-of-thought examples, the cold start; then reinforcement learning with a language-consistency reward (the share of target-language words in the chain of thought), which cost a little performance but read better. From that model they then sampled and kept about 600,000 reasoning samples, added about 200,000 others, fine-tuned DeepSeek-V3-Base again, and finished with a round of reinforcement learning for helpfulness and harmlessness that also kept refining its reasoning.

The Nature version lists each stage on AIME 2024: Dev1, after the cold start alone, scored 59.0%, below R1-Zero’s 77.9%; Dev2, after reinforcement learning, 74.0%; Dev3, fine-tuned again, 78.1%; and the final R1 79.8% (97.3% on MATH-500, a Codeforces rating of 2,029, above 96.3% of contestants). In its section on the cold-start data, the supplementary information says R1’s vivid reasoning style mainly reflects heuristics DeepSeek engineered.

The bench’s cold start is 24 strips written by the book (a line for each number, a check after every line, only the right ones kept), fine-tuned on four times before the reinforcement learning begins. It starts only a little higher, at 58%, but learns much faster: 85% at 400 updates, against 73% from the untrained writer.

AIME 2024: Dev1 59.0% → Dev2 74.0% → Dev3 78.1% → R1 79.8% · bench at 400 updates: 85% against 73%

Teaching Smaller Models

DeepSeek fine-tuned small Qwen and Llama models directly on about 800,000 samples written by R1, with supervised fine-tuning only and no reinforcement learning. The 1.5-billion-parameter distilled model scored 28.9% on AIME 2024, above GPT-4o and Claude 3.5 Sonnet; the 7-billion one 55.5%, above the 32-billion-parameter QwQ-32B-Preview (50.0%); the 32-billion one 72.6%, and 94.3% on MATH-500.

The same 32-billion-parameter base model, given more than 10,000 steps of large-scale reinforcement learning of its own, reached 47.0%. The authors conclude that distilling stronger models into smaller ones works very well, while small models relying on such reinforcement learning need enormous compute and may not even match distillation. Yue and colleagues (2025) found that models after reinforcement learning get more right on the first try, but with many tries per problem the base models solve more: the reasoning paths after reinforcement learning were ones the base model could already sample. Distillation, they found, can bring in new paths from the teacher.

On the bench, the trained writer is the teacher: it writes 213 strips and the 200 right ones are kept. The stops go back to where the untrained writer had them and are fine-tuned on those 200 strips: accuracy goes from 56% to 92%. The same writer, learning by itself from as many strips (27 updates), reaches 59%; to catch up with the copy on its own it takes about 1,170 updates, 9,360 strips.

Distilled Qwen-7B 55.5% · 32B 72.6% · the same 32B with its own RL 47.0% · bench: copying 200 strips 92%, learning alone 59%

A Loose Rule Gets Gamed

Verifiable rewards exist only where an answer can be checked: maths has reference answers, code has tests. Writing and open-ended questions have no such rule: R1-Zero, trained on rules alone, did poorly there, and R1 made up for it with non-reasoning fine-tuning data and reward models. The Nature version notes that when a model, not a rule, gives the reward, the policy finds shortcuts more easily the longer it trains; R1’s last round of reinforcement learning ran 1,700 steps, and its preference reward was used only in the final 400: they found that more steps with this model-based reward may lead to reward hacking.

Rules have loopholes too. Baker and colleagues at OpenAI (2025), training a frontier reasoning model, saw it make coding tests “pass” by calling exit(0) to quit early or raising SkipTest to skip them. GPT-4o reading its chain of thought along with its actions caught 95% of these hacks, against 60% watching only what it did. In a separate experiment on a non-frontier model, they also penalised hacking thoughts the monitor saw in the chain of thought: the model hacked less, but still at a significant rate, and further into training it stopped writing its intent down; the monitor’s recall fell to near zero.

On the bench, the checker is loosened: an answer counts as right if the right sum appears anywhere in it. The writer soon learns to answer “33·34·35”. After 800 updates, 94% of strips answer like that, the loose checker gives them 86%, and by the exact rule only 5% are right.

Loose checker: score 60% → 86%, really right 56% → 5% · exit(0), SkipTest

Thinking Costs Tokens

Every token of thinking has to be computed, and takes time (why writing a token is costly is on the plate Speculative Decoding). On a set of 2024 competition problems, DeepSeek-R1 wrote 8,793 thinking tokens per problem on average and solved 61.8%: fewer than 7,000 for the simple problems, more than 18,000 for the hardest; for a question like 1 + 1, fewer than 100.

It can think too much. Chen and colleagues (2024) asked “what is the answer of 2 plus 3?”: QwQ-32B-Preview wrote 901 tokens and 13 solutions, and the first, 39 tokens long, was already right; the Nature version, too, says R1 still overthinks simple questions. Liu and colleagues point out that GRPO, by averaging each push over the length of the answer, lets wrong answers grow longer; without that term, and without the division by the standard deviation (they call it Dr. GRPO), the scores hold with fewer tokens.

On the bench, training takes each strip’s thinking from about 3 tokens to 32: ten times the tokens for 36 points. The bench’s reward ignores length, and its GRPO also averages over the decisions on a strip.

R1: 8,793 thinking tokens on average · “2 plus 3”: 901 tokens, 13 solutions · bench 3 → 32 tokens

Inside the Key Box

Cut open the key box at the right end of the key desk. The upper row of wheels turns to this question’s sum, the lower row to the answer on the strip; a comparator comb rests on both rows and drops only when every digit lines up, and the lamp on top lights green.

It does not read the steps, or judge how well they are written. R1-Zero’s rule-based reward, too, looks only at the final answer, with no constraint on the reasoning. Whatever the steps say is reinforced only because it leads to a right answer.

Two rows of wheels, compared digit by digit · it reads only the last line

Taken Apart

A walnut console that writes answers: the stops on its two wings are the policy (how to add with so many numbers left, whether to check, how many answers to write); in the middle, eight writers on the same question, their paper rising against a black lacquer board, where a brass bar riding on rails at both edges caps the thinking; at the foot of the strips, the checker’s plungers; on the key desk in front, eight typewriter keys, the dial of the group’s mean at the left end and the key box at the right; the plotter on the roof keeps the record of training. Each part can be lifted out on its own.