The Mixture of Experts

Many parameters, little work

DeepSeek-V3, released at the end of 2024, has 671 billion parameters and uses 37 billion of them on each token, about 5.5%. Mixtral 8x7B has 46.7 billion and uses 12.9 billion per token.

The model on the table is far smaller: 16 experts, 53,030 parameters, of which each letter uses 8,902, about 17%. The experts not chosen stand idle, their lamps dark.

This page’s model: 16 experts · 2 chosen per letter · 8,902 of 53,030 parameters used

The router opens two lines

Each letter first stops on the turntable in the middle. The router is a 48 × 16 table: the letter’s 48 numbers times the table give 16 scores, and a softmax turns them into 16 probabilities. The 16 pins round the turntable stand as tall as those probabilities.

Only the top two are kept. A second softmax over those two scores gives two weights that add up to 1. The letter goes to those two experts, and the two results are added back with those weights. That is how Mixtral does it; the sparse gate of 2017 also added a little learned noise to the scores, and the Switch Transformer of 2021 keeps only one.

scores = letter × router table → keep the top k → weights = softmax of those k scores

The rest do nothing

An expert that is not chosen computes nothing for that letter. So the work per letter depends on how many experts are chosen and how big one expert is, not on how many there are.

Mixtral has 8 experts in each layer and picks 2 per token. Of its 46.7 billion parameters, each token uses 12.9 billion: attention and the vocabulary are shared by all, and a quarter of the experts run.

Mixtral 8x7B: 2 of 8 · 46.7 billion in all · 12.9 billion per token

More experts, better guesses

The same training, and each letter still runs just two experts. Going from 4 experts to 8, 16 and 32 leaves the work per letter almost unchanged while the parameters grow from 15 thousand to 100 thousand. On 100,000 letters the models never saw in training, the average cost of guessing the next letter (cross-entropy, lower is better) falls from 2.47 bits to 2.20.

The control is one ordinary feed-forward block, twice as wide as an expert, so it costs the same as two experts: 2.60 bits, right 45.7% of the time; 16 experts are right 50.6%. In 2021, Switch-Base had 7 billion parameters and the same compute per sequence as the 0.2-billion-parameter T5-Base; its 64-expert version reached the same quality in one seventh of the time.

Cross-entropy: dense 2.60 · 4 experts 2.47 · 8: 2.36 · 16: 2.27 · 32: 2.20 bits

Inside an expert

Open any expert and there is an ordinary feed-forward block: 48 numbers become 32, negatives become 0, and they become 48 again. The 32 steel leaves in the cabinet are those 32 middle values; a lit leaf rises as high as its value for the letter now inside.

Every Transformer layer already has one such block (see the attention plate). A mixture of experts replaces it with many side by side and lets only a few work; attention stays shared. GShard, in 2020, did this in every other layer.

One expert: 48 → 32 (ReLU) → 48 · 3,152 parameters

The rich get richer

The router trains together with the experts. The 2017 paper that introduced the sparse gate found that it tends to settle on always giving large weights to the same few experts, and that this feeds itself: the favoured experts train faster and so get chosen even more.

This page trained one version with no balancing at all. On 100,000 letters, expert 1 takes 18.3% of the work and expert 9 is never chosen, its 3,152 parameters taking up room for nothing. The glass tubes beside the cabinet doors count what each expert takes in every 32 letters.

No balancing: the busiest expert 18.3% · one at 0% · coefficient of variation 0.61

A small penalty

The usual remedy is one more term in the training objective. The Switch Transformer writes it α · N · Σ fᵢ · Pᵢ: fᵢ is the share of tokens actually sent to expert i, Pᵢ the router’s average probability for it. The more even the routing, the smaller the term. The paper uses α = 0.01.

The 16-expert model on the table carries this term. The busiest expert takes 10.4%, the quietest 3.7%, a coefficient of variation of 0.30. The guessing barely changes: 2.27 bits either way.

α · N · Σ fᵢPᵢ, α = 0.01 · busiest 10.4% · quietest 3.7% · coefficient of variation 0.30

Full, so sent back

Spread over many chips, each expert must know in advance how many tokens it will take per batch: its capacity, the tokens in the batch divided by the number of experts, times the experts per token, times a capacity factor. Tokens past the capacity get no work done in that layer; they carry on unchanged along the residual connection. They are said to be dropped.

Here every 32 letters make a batch, and at a factor of 1.0 each expert takes 4: the red collar marks that line. At 1.0, a fifth of the deliveries come back; at 1.25 about a tenth; at 2.0 only 0.4%, but every expert keeps twice the room. In the Switch Transformer’s runs, usually fewer than 1% were dropped; DeepSeek-V3 balances well enough to drop none, in training or in use.

Batches of 32 letters · factor 1.0: capacity 4, 19.8% sent back · 1.25: 10.2% · 2.0: 0.4%

What do experts know?

The name suggests one expert for maths and one for history. The truth is plainer. The Mixtral paper compared arXiv papers, biology abstracts and philosophy, and found each expert chosen about equally often in all of them: no sign of division by topic. What it did find was syntax: ‘self’ in Python, or the indentation in code, often goes to the same expert.

The small model on the table is the same, its division of labour about the shapes of letters. More than eight in ten of the letters experts 10 and 14 take come right after a space, at the start of a word; expert 2 mostly takes what follows a vowel, half the time an a; expert 13 most often takes the letter after t.

Expert 10: 88.6% after a space · expert 2: 82.4% after a vowel · over 100,000 letters

Shared experts and a bias

DeepSeek-V3 cuts its experts finer: 256 routed experts in each layer, each with only 2,048 middle values, 8 chosen per token, plus 1 shared expert that every token passes through, to hold what all of them need. Of its 61 layers, the first 3 have no experts.

It balances almost without a penalty term. Each expert has a bias that is added to its score only for choosing, never to its weight. After every training step an overloaded expert’s bias drops by γ and an underloaded one’s rises by γ (γ = 0.001). The version on the table trained this way keeps every expert between 5.8% and 6.8% of the work, a coefficient of variation of 0.035, and guesses as well as the penalised one.

DeepSeek-V3: 1 shared + 8 of 256 · bias balancing · this page: shares 5.8–6.8%

Only compute is saved

Every expert has to sit in memory, ready to be chosen. The Mixtral paper says its memory cost is proportional to its 47 billion parameters: about 93 GB at 2 bytes each, while each token costs the compute of a 12.9-billion-parameter model.

Many experts must be spread over many chips, and tokens have to travel between them. GShard trained its model of 2,048 experts per layer on 2,048 TPU cores; DeepSeek-V3 sends each token to at most 4 machines to keep the traffic down. Training is also less stable: the Switch Transformer computes its router in 32-bit floats and starts its weights ten times smaller.

Memory for every parameter · compute for the used ones · traffic between chips · unstable routing

Taken apart

The turntable is the router. An expert’s cabinet is a body, a roof, a work lamp, a feed drum and a load tube; each part can be lifted out on its own.