Learning from Human Feedback

Smaller, yet preferred

InstructGPT, 2022: on the prompts collected for it, labellers preferred the answers of its 1.3-billion-parameter version to those of the 175-billion-parameter GPT-3. The two share an architecture; InstructGPT had been tuned further on human data. At 175 billion parameters each, the tuned model’s answers were preferred 85 ± 3% of the time.

The tuning has three steps: supervised fine-tuning on demonstrations people wrote; a reward model trained on people’s comparisons; then reinforcement learning against the reward model. At 175 billion parameters, the supervised fine-tuning and reinforcement learning runs together took about 65 petaflop/s-days (not counting the reward model), under 2% of the 3,640 that pretraining GPT-3 took. ChatGPT was trained with the same methods, with slight differences in how the data were collected.

InstructGPT · 1.3B parameters preferred to the 175B GPT-3 · at 175B each, 85 ± 3%

First, copy the demonstrations

Pretraining teaches a model to guess the next word on a web page, which is not the same as answering what a user asks. The small model on the bench can say only twelve things, all to one question: “Why is the sky blue?” Each card’s height is how often it says that answer, on a log scale from 0.1% to 100%.

After pretraining, 44% of the time it carries on with more questions. Supervised fine-tuning is gradient descent on demonstrations written by labellers: 12 of them, 80 steps, and “more questions” falls to 9% while the answers rise. A red clamp marks where each card ends up; everything after is measured from there. InstructGPT’s demonstrations came from about 13,000 prompts.

Supervised fine-tuning · 12 demonstrations · 80 steps · “more questions” 44% → 9%

Two answers, pick one

In the second step people stop writing and start comparing. InstructGPT’s labellers, about 40 of them, ranked 4 to 9 answers at a time from best to worst, and every pair in a ranking counted as one comparison. On the same pair, labellers agreed with each other about 73% of the time.

Why not ask for scores? Christiano and colleagues found in 2017 that on some tasks it was much easier for people to give consistent comparisons than consistent scores. A score asks everyone to share one scale; a choice between two only asks which is better. The balance weighs two cards at a time; the red seal is the labeller’s choice.

InstructGPT · 4 to 9 answers ranked at a time · training labellers agree 72.6 ± 1.5%

Only the difference counts

How do choices become scores? Through the paired-comparison model of Bradley and Terry, 1952, in today’s notation: each answer has a score r, and A beats B with probability e^r_A / (e^r_A + e^r_B), which is σ(r_A − r_B). A difference of 0 is even odds; 1 gives 73%; 3 gives 95%.

It works like a chess rating: only the difference says who is likelier to win. Add the same number to both scores and nothing changes, like adding the same weight to both pans. So a reward model’s scores have no zero, only differences. The balance’s pointer swings 2σ(Δr) − 1 of its full travel.

P(A beats B) = σ(r_A − r_B) · a difference of 1 → 73% · of 3 → 95%

A model that learns to score

The page made up a labeller: he privately scores the twelve answers, high for those that explain, low for those that repeat themselves. He compared 120 pairs, each time choosing with odds σ(difference), so now and then he picks the worse one. All of this is generated by the page, not human data.

The reward model on the bench is crude; it sees three things: whether an answer answers, how many sentences it has, and whether it flatters. Fitting Bradley–Terry to the 120 choices, it learns +3.44 for answering, +0.16 a sentence and −1.24 for flattery. On the answers the SFT model usually gives, it agrees with the labeller 77% of the time; a judge who knew his scores would reach only 86%. The weight behind each card is its score. InstructGPT’s reward model had 6 billion parameters.

Reward model · 120 pairs · r = 3.44 · answers + 0.16 · sentences − 1.24 · flattery · agrees 77%

Training against the score

The third step is reinforcement learning. InstructGPT set it up as an environment of one move: a prompt comes in, the model writes one answer, the reward model scores it, and the episode ends. PPO, proximal policy optimisation, samples in batches: answers that score better than expected become a little more likely, those that score worse a little less. Real PPO learns what to expect with a value network; here, expected is the batch’s average.

PPO stops rewarding any change that would push the ratio of new to old probability past 0.8–1.2 (ε = 0.2), so no step has a reason to go far. Here each step samples 128 answers and takes 4 gradient passes over them. On the drum recorder at the front right, the red pen writes the reward model’s average score and the dark pen the labeller’s.

PPO · 128 answers a step · no credit for pushing π / π_old past 0.8–1.2 · β = 1

A spring

Each answer’s score also loses β · log(π / π_SFT): the more often the model gives it than the SFT model did, the more it loses. On average that comes to β times the KL divergence, how far the new model has moved from the SFT model. The 2020 summarisation study gave it two jobs: keep the model from collapsing onto one answer, and keep it from writing things too different from what the reward model saw in training.

The best policy for this objective is π ∝ π_SFT · e^(r/β). On the log scale, each card sits above its red clamp by its score over β, less one number shared by all the cards. β is the spring’s stiffness: at β = 4 the cards hardly move; at β = 0.1 the top-scoring card rises to 99%.

π ∝ π_SFT · e^(r/β) · β = 4 → 1 → 0.3 → 0.1

Higher scores, worse answers

Lower β to 0.02 and train again. Within a dozen steps the model writes card 8 almost every time (the one with the fourth power and the sunset), and the labeller’s score climbs from 4.1 to 6.4. Some two hundred steps in, it finds card 11: the same content and five more sentences of repetition. The reward model pays for sentences, 0.8 for those five, so the model writes nothing else. The red pen keeps rising; the dark pen falls to 2.6, below where SFT left it.

The fault is the reward model’s: it cannot see repetition, only count sentences, and the SFT model seldom wrote that long; card 11 came up in only 3 of the 120 pairs. Given 20,000 pairs it still prefers that card, because what it looks at stays the same. When ChatGPT came out, OpenAI wrote that it was often excessively verbose, because of biases in the training data (trainers prefer longer answers that look more comprehensive) and “well-known over-optimization issues”.

β = 0.02 · reward model 3.46 → 4.42 → 5.23 · labeller 4.13 → 6.37 → 2.6

Over-optimisation has a shape

In 2022 Gao and colleagues measured it. A 6-billion-parameter reward model from InstructGPT played the people; its judgements trained proxy reward models of 3 million to 3 billion parameters, and policies were optimised against the proxies. The proxy score kept climbing; the stand-in people’s score rose and then fell. Against d = √KL, reinforcement learning followed d(α_RL − β_RL log d) (both fitted coefficients, not the KL coefficient), and β_RL changed smoothly with the proxy’s size.

They also found that in their setting a KL penalty acted like stopping early: it did not raise the true score at a given distance. They add that this may be sensitive to hyperparameters. In the earlier summarisation study, the harder the optimisation, the further the reward model’s predictions parted from people’s preferences, until they ran the other way.

Gao et al. 2022 · RL: R(d) = d(α_RL − β_RL log d) · d = √KL · “people” = a 6B reward model

The rack cut open

Cut away the front of the rack and what hides behind the cards shows: each carriage rides two guide rods; a spring runs from it to the red clamp behind; above, a cord runs over a pulley to a hanging weight. The weight is the reward model’s score; the further the spring stretches, the harder it pulls back.

Where the two balance is the answer on the page before: log π − log π_SFT = (r − c) / β, where c is the one number shared by all the cards that keeps the probabilities adding up to 1. Hang equal weights on every cord and nothing moves: adding the same number changes nothing.

log π − log π_SFT = (r − c) / β · weight = score · spring stiffness = β

No separate reward model

In 2023 Rafailov and colleagues turned that best policy around: r = β log(π / π_ref) plus a number that depends only on the question. Put that into Bradley–Terry and that number cancels, so comparisons can train the model itself. This is direct preference optimisation, DPO: a classification-style loss, no separate reward network, no sampling from the model during training and no reinforcement learning.

On the bench, the same 120 pairs, the weights taken off, and the cards go straight to their places. In this small example it lands exactly where the exact optimum against the reward model does. In practice, with limited data and imperfect training, neither reaches the exact optimum; in the DPO paper’s own tests, on models of up to 6 billion parameters, the two gave different results.

DPO · loss = −log σ(β log π/π_ref (chosen) − β log π/π_ref (rejected)) · β = 1

Whose score

What RLHF optimises is a score: the preferences of a particular group of people under particular instructions, as a model has learned them. InstructGPT’s authors wrote that they aligned the model to “the stated preferences of a specific group of people (mostly our labelers and researchers)” rather than any broader notion of human values, and that their models “are neither fully aligned nor fully safe” and still make up facts.

The comparing can be handed to a model too. In Anthropic’s Constitutional AI (2022), for harmlessness, a model critiques and revises its own answers, and another model compares pairs, both against written principles (16 for each stage). Reinforcement learning on those AI comparisons is called RLAIF. For helpfulness it still used 135,296 human comparisons.

InstructGPT · about 40 labellers · Constitutional AI · 16 principles + 135,296 human comparisons

Taken apart

Three instruments on one bench: the balance in its glass case (the reward model), the twelve-column rack (the model’s policy), and in front the card tray and the two-pen drum recorder. Each can be lifted out on its own.