Sparse Autoencoders

A Feature for a Bridge

In 2024, researchers at Anthropic trained a sparse autoencoder on a middle layer of Claude 3 Sonnet, their medium-sized model: a “dictionary” of about 34 million entries, of which about 12 million were alive. One of them (34M/31164353) answers to the Golden Gate Bridge: text about the bridge, the first sentence of its Wikipedia article in several languages, even pictures of it, although the dictionary was trained on text alone.

The researchers clamped that entry at ten times its largest value in the training data, at every word, and asked the model “what is your physical form?” It answered: “I am the Golden Gate Bridge, a famous suspension bridge that spans the San Francisco Bay…” Unclamped, it said it had no physical form. That May, a version of Claude with this feature turned up was open to the public for 24 hours. The bench is the same method at small scale, all computed in your browser.

Claude 3 Sonnet · a dictionary of about 34 million · Golden Gate Bridge 34M/31164353 · clamped at 10× its maximum

Six Things in Two Numbers

First, the model being read. The box on the left is its first half: six inputs, which we call rain, flower, bridge, lamp, leaf and sea so they are easy to follow. To the model they are just six numbers, each present one time in twenty. It squeezes them into two numbers; the bright dot on the oscilloscope is those two. The box on the right reads the six back out of them.

Training gave each thing a direction, about 60 degrees from its neighbours: six directions squeezed into one plane. There are more features than dimensions because they seldom turn up together. This is called superposition (the plate More Features Than Neurons shows why it works). So neither number stands for one thing: from the horizontal one alone, you can’t tell a bridge from a lamp.

Six things → two numbers → six things · neighbouring directions about 60° apart

A Longer Dictionary

The sparse autoencoder sees only the two numbers. It is never told there are six things. It has two halves. The encoder spreads the two numbers over eight “lines”, each a learned direction that glows brighter the closer the input points along it. The decoder adds the glowing lines back, each along its own direction, to rebuild the two numbers. Training asks for two things only: a close rebuild, and few lines lit at once.

On the bench, the first prism splits one beam into eight and sends them to a frame. The frame has 16 rods; each of the eight lines takes one, and the place of the cursor on its rod is that line’s direction; the marks on the frame are the six things’ true directions (this is a toy, so we know the answer). A second prism joins the light back into one beam. Real dictionaries are usually far wider than the layer: in 2023, Anthropic decomposed the activations of the 512 neurons in a one-layer model’s MLP with dictionaries from 512 entries (one per neuron) to 131,072 (256 per neuron), and studied one of 4,096 most closely.

Encoder f = TopK(E(h − c) + e) · decoder ĥ = D f + c · 2 numbers → 8 lines

One Line at a Time

What keeps it sparse? The bench uses the rule OpenAI chose in 2024 (a k-sparse autoencoder, first described in 2013): for each input, keep only the k brightest lines and set the rest to zero. With k = 4, an input that holds something lights 2.8 lines on average, every line answers to two or three things, and the cursors sit between the marks. With k = 1, an input lights one line: trained again from the same start, six cursors each slide onto a mark, all within 2 degrees, and the other two stop between marks.

In 2023 and 2024 Anthropic used another way: a penalty on the lines’ total brightness (L1). It works too, but dims every lit line a little; keeping k lines dims nothing, and sets how many light directly. In this toy of two numbers, where neighbours are only 60 degrees apart, a small L1 penalty never pulls the lines apart, and one large enough to do it loses more than half the variance.

k = 4 · 2.8 lines on average, none answers to one thing · k = 1 · 6 lines on the marks, all within 2°

Naming a Line

Lines come without names. To learn what one stands for, run a large batch of inputs through (4,096 here) and pick the ones that make it brightest. The line whose cursor sits in the middle has five brightest inputs that all hold the bridge (the very brightest holds a lamp, a flower and the bridge), so we call it “bridge”. Here we can check the answer: its direction is 0.45 degrees from the bridge’s.

Real models come with no answer to check. In 2023, one of the authors, scoring blind, rated the activation ranges of neurons and features on a rubric out of 14: the median feature interval scored 12, the median neuron 0. The researchers also had Claude read examples from across a feature’s range, strongest to weakest, write an explanation, and use it to guess the activations on examples it had not seen. But a name is a reading: in 2024 Anthropic wrote that it is also plausible its interpretations “slightly misrepresent the feature’s actual function”.

The bridge line · its 5 brightest inputs all hold a bridge · 0.45° from the bridge’s direction

Clamping a Line

A name is a guess, and clamping is a way to test it. With no key pressed, hold the bridge line at half its largest value (2.04): the decoder adds that line’s direction to the two numbers, and the bridge meter on the right reads 0.78, while its neighbours lamp and flower show only 0.17 and 0.20. Real experiments work the same way: only the autoencoder’s rebuilt part is changed, the error is kept as it is, and the clamp holds at every word.

Clamp it at its largest value and the bridge needle hits the end stop (2.00), while lamp and flower rise to 0.78 and 0.84: in two numbers, a bridge pushed hard looks like a lamp and a flower too. The other way round, press the bridge key (0.9) and clamp the line to 0: the bridge reading falls from 0.84 to 0. Anthropic writes that clamping too far (say ±100 times the maximum) usually makes the model talk nonsense; they mostly tried values between −10 and 10 times.

Bridge line clamped · 0.5× its maximum → bridge 0.78 · 1× → bridge 2.00, lamp 0.78, flower 0.84

From Eight Lines to Millions

The same method, at far larger scale. In 2023, Cunningham and colleagues trained sparse autoencoders on the residual streams of Pythia-70M and Pythia-410M, with GPT-4 writing explanations and GPT-3.5 predicting activations from them: in Pythia-70M their features were easier to explain than the directions PCA and ICA found, though the lead shrank in later layers. In 2024, Anthropic trained three dictionaries for Claude 3 Sonnet, of about 1, 4 and 34 million entries, with fewer than 300 lit per word on average; OpenAI trained one of 16 million on GPT-4, over 40 billion tokens.

Longer dictionaries split lines. In the one-layer model, going from 512 to 4,096 to 16,384 entries, the lines for base64 strings went from 1 to 3 to many more: one prefers letters, one digits, one base64 that encodes plain text. On the bench, at 16 lines, the six things still get one line each; of the extra lines, 5 become pair lines and 5 never light.

Claude 3 Sonnet · about 1M / 4M / 34M · fewer than 300 lit per word · GPT-4 · 16 million

Lines Without a Name

Of the eight lines, only six stand for one thing each. The other two light only when two things turn up together: one’s brightest examples are flower with rain and flower with sea, the other’s a lamp with a leaf. In 4,096 inputs they lit 11 and 6 times, and no one name fits either. The lamp line is odd in another way: on inputs that hold nothing it still glows a little (0.005), so it lit on 77% of all inputs.

Real dictionaries also have lines that stand for nothing. In Anthropic’s 2023 dictionary of 4,096 lines, 168 never lit on 100 million examples (“dead”), and 292 more lit less than once in a million; in the 2024 dictionary of 34 million, about 65% were dead. In OpenAI’s test runs without remedies, up to 90% died; starting the encoder as the decoder’s transpose and adding an auxiliary loss left only 7% dead in their 16-million dictionary. The bench’s encoder also starts as the decoder’s transpose.

8 lines: 6 for one thing each, 2 pair lines, 1 nearly always on · in Claude 3 Sonnet’s 34 million, about 65% dead

What the Rebuild Misses

The two numbers the decoder rebuilds are not quite the original ones. On the bench, 1.3% of the variance is lost, almost all of it (98%) on inputs that hold two or more things: with one line lit at a time, it can name only one thing. What is missing goes round by the yellow fibre, and when a line is clamped it is added back unchanged.

In real models the gap is wider. Claude 3 Sonnet’s three dictionaries rebuilt at least 65% of the variance, so up to 35% was left unexplained. Put OpenAI’s 16-million rebuild into GPT-4 and its language-modelling loss matches a model trained with 10% of GPT-4’s pretraining compute. Cunningham and colleagues replaced Pythia-70M’s layer-2 residual stream with the rebuild, and perplexity rose from 25 to 40.

Bench: 1.3% of the variance lost · Claude 3 Sonnet: up to 35% lost · GPT-4 with the rebuild ≈ 10% of the compute

Found Is Not Understood

A line that lights shows it varies with something, not what the model does with it. Anthropic writes that dataset examples “do not establish causal links to model behaviors”: a feature could always respond to something and have no function. In Claude 3 Sonnet they found features to do with unsafe code, sycophancy and deception, and wrote that the work “does not show that any features are actually useful for safety”.

The dictionary is far from complete, too: Claude 3 Sonnet can list all of London’s boroughs, yet the 34-million dictionary had features for only about 60% of them, and the authors think they are quite likely orders of magnitude short of “all the features”. The training goal (a close rebuild, few lit) is itself only a stand-in for being interpretable. In OpenAI’s words, finding features at one point in the model is only one step towards interpreting it.

Inside the Meters

Cut open the box on the right: behind each meter sit a coil and a hairspring. The two numbers drive the coil; the hairspring is wound up in advance and holds the needle against the zero stop, so it moves only once the push passes about 0.44. That is the model’s last step, ReLU(Wᵀh + b), with b about −0.44: small leaks never reach the needle.

It is also why a clamp lights the neighbours. Lamp and flower point about 60 degrees from the bridge, so each gets half of the bridge’s push; push the bridge hard enough and that half clears the stop. Sea, opposite the bridge, gets a negative push, and its needle is held down harder.

x′ = ReLU(Wᵀh + b) · b ≈ −0.44 · a push under 0.44 leaves the needle still

Taken Apart

One optical table: on the left the keys and the oscilloscope (the model’s first half), in the middle the splitting rig (the sparse autoencoder: two prisms and a frame of 16 rods), on the right the six meters (the model’s second half), and a yellow fibre carrying the error round. Each piece can be lifted out on its own.