# 16,384 GPUs, One Model

Laplace’s Mini World · Plate F68 · AI

> Training Llama 3 405B: each layer split over 8 GPUs, the 126 layers cut into 16 stages, and that copy repeated 128 times on different data: 16,384 GPUs.

- Interactive model: https://world.lapla.si/en/cluster/
- Embed: https://world.lapla.si/embed/en/cluster/
- 中文: https://world.lapla.si/zh/cluster/
- Updated: 2026-11-05

## 01 128 copies of 128

To train the largest model of its Llama 3 family, Meta used up to 16,384 NVIDIA H100 GPUs at once. The model has 405 billion parameters; it read 15.6 trillion tokens and took about 3.8 × 10²⁵ floating-point operations. The paper says “up to 16K”; its table lists runs on 8,192 and on 16,384 GPUs.

The GPUs sit in Meta’s own servers, eight to a server and two servers to a rack. On the plinth is a row of such racks: 8 racks, 16 servers, 128 GPUs, with a network rack at the end. In Llama 3’s layout one copy of the model spans exactly 128 GPUs, so 16,384 GPUs hold 128 copies. Whether the 128 GPUs of a copy stand side by side the paper does not say; this row is the page’s drawing.

Parallelism does not make a model smarter. The amount of computing is the same; spreading it over more GPUs gets it done in months instead of thousands of years, and the price is that the GPUs must keep passing data to each other.

*up to 16,384 H100s · 405 billion parameters · 15.6 trillion tokens · about 3.8 × 10²⁵ operations*

## 02 Too much for one

One H100 holds 80 GB in the HBM3 memory stacks on either side of its chip. The 405 billion parameters alone, at 2 bytes each, take 810 GB: more than ten GPUs’ worth.

Training also keeps the gradients and the optimiser’s state. By the ZeRO paper’s accounting for mixed-precision Adam, that is 16 bytes a parameter: 2 each for the weights and the gradients, 12 for a 32-bit copy of the weights, the momentum and the variance. For 405 billion parameters that is 6.48 TB, at least 81 GPUs, before counting the intermediate results each layer produces (the activations). Meta does not publish how many bytes Llama 3 used per parameter; this is the usual accounting (this page’s arithmetic).

Nor is there time. At the paper’s real speed of about 400 trillion operations a second per GPU, 3.8 × 10²⁵ operations would take one GPU about 3,000 years; 16,384 GPUs together, about 67 days (computing only, no interruptions; this page’s arithmetic).

*weights 810 GB · training state about 6.48 TB · one GPU 80 GB · about 3,000 years on one*

## 03 Eight in a box

Meta’s server is called Grand Teton: 380 mm tall, 537 mm wide, 805 mm deep, and 118.7 kg when full. It has three trays: two CPUs at the top; a switch tray in the middle with eight 400G network cards, one for each GPU; the GPU tray at the bottom. In the front server on the plinth that tray is pulled out, and both servers in that rack are cut open at the side.

The GPU tray holds NVIDIA’s HGX board: eight H100s and four NVSwitch chips. Every GPU is wired to all four NVSwitches, and every NVSwitch to all eight GPUs. Each GPU has 18 NVLink links of 25 GB/s each way: 450 GB/s each way in all. NVIDIA’s 900 GB/s counts both directions.

Each GPU in the Llama 3 run had a 700 W power limit, and the servers are air-cooled: a tall heat sink on every GPU, 16 fans at the back of the box; cool air comes in at the front and leaves hot at the back. How data moves between memory and the chip inside one GPU is the subject of the plate “FlashAttention and GPU Memory”.

*8 H100s · 4 NVSwitches · 450 GB/s each way · 700 W each · air-cooled*

## 04 Between the racks

Each server’s eight network cards are cabled in copper to the switch at the top of the rack (Meta uses the Minipack2). A rack holds two servers, 16 GPUs. From the rack switch, single-mode fibre takes over; on the plinth it runs along the yellow trough overhead to the cluster switches at the row’s end. Each card carries 400 Gb/s, which is 50 GB/s each way: a ninth of NVLink.

Llama 3’s network has three layers. 192 racks are joined by cluster switches into a pod of 3,072 GPUs, with full bandwidth between any two halves of it; eight pods are joined by aggregation switches into a cluster of 24,576 GPUs, and that top layer has a seventh of full bandwidth (1:7). So the way the work is split, and where the job runs, keep traffic between pods low.

Google links thousands of TPUs another way, with optical circuit switches: see the plate “The TPU and Its Systolic Array”.

*one 400 Gb/s card per GPU · 16 GPUs a rack · 3,072 a pod · 24,576 in all · 1:7 between pods*

## 05 Same model, different data

The simplest split: each copy of the model computes all of it, but on a different batch of data. After the backward pass each copy holds gradients from its own data; the copies average them and all take the same step, so they stay one model. This is data parallelism.

On 16,384 GPUs, Llama 3 ran 128 copies, reading about 16.8 million tokens a step: 131,072 per copy (16 sequences of 8,192 tokens). It used fully sharded data parallelism (FSDP): the optimiser’s state and the gradients are also shared out among the copies, so no copy keeps a full set. By the ZeRO paper, sharding the optimiser’s state alone cuts each GPU’s memory for the model’s state to about a quarter when there are many copies, at no extra traffic.

*128 copies · about 16.8 million tokens a step · 131,072 per copy · gradients averaged every step*

## 06 Round the ring

The usual way to average the gradients is a ring all-reduce. N GPUs form a ring and each cuts its gradient into N pieces. For N − 1 steps each passes a piece to the next GPU, which adds its own; at the end each GPU holds one piece fully summed. For N − 1 more steps the summed pieces go round until every GPU has them all.

Each GPU sends 2(N − 1)/N times the gradient’s size: 1.75 times for 8 GPUs, 1.98 times for 128, so it barely grows with N. Patarasuk and Yuan proved in 2009 that without compressing the data, whatever the algorithm, some GPU must send at least that much. What grows with N is the number of steps: a bigger ring means more waiting.

So the time depends mainly on the links. For a 10 GB gradient round a ring of 128, each GPU sends about 19.8 GB: about 0.4 s over the 50 GB/s network, about 0.044 s over 450 GB/s NVLink (bandwidth only; this page’s arithmetic). In training this traffic runs alongside the computing as far as possible.

*each GPU sends 2(N − 1)/N times · 8 GPUs: 1.75 · 128: 1.98 · 2(N − 1) steps*

## 07 One layer in eight pieces

Each layer of the model is a few large matrix multiplications. Tensor parallelism cuts a layer’s matrices and shares them among several GPUs. The Megatron-LM split of 2019: the feed-forward block’s first matrix by columns, the second by rows, with no communication in between; attention is shared out by heads.

Then every layer needs two all-reduces in the forward pass and two in the backward pass: after each layer the GPUs stop to settle up. So tensor parallelism stays among the eight GPUs of one server, on NVLink; in every Llama 3 layout it is 8-way. Llama 3 405B’s layers are 16,384 wide, and each GPU holds an eighth of every matrix.

*2 all-reduces forward, 2 backward · kept inside one server · Llama 3: 8-way*

## 08 An assembly line

Pipeline parallelism cuts by layers: Llama 3 405B’s 126 layers go into 16 stages of about 8 layers, each on its own group of GPUs. On the plinth each of the row’s 16 servers is drawn as one stage: the first computes the first layers and passes its results (the activations) to the second, and so on to the last; in the backward pass the gradients come back the other way.

With one batch, only one stage would work at a time. So the batch is cut into many micro-batches, like parts on an assembly line: when the first micro-batch moves on to the second stage, the first stage starts on the second one. To keep the stages equally busy, Llama 3’s first and last stages each hold one layer fewer: they also do the embedding and the output.

*126 layers · 16 stages · about 8 layers each · activations forward, gradients back*

## 09 The bubble

At the start of the pipeline the later stages wait for the first micro-batch; at the end the earlier stages wait for the last gradients. That idle time is called the bubble. With every micro-batch going forward first and then all coming back (GPipe’s schedule, as drawn here), each GPU is idle for (p − 1)/(m + p − 1) of the step, with p stages and m micro-batches.

With 16 stages and 16 micro-batches that is almost half the time (48 %); with 64 it is 19 %, with 128 it is 10.5 %. The GPipe paper found the bubble negligible once there were at least four times as many micro-batches as stages; by the formula, 16 stages and 64 micro-batches still leave 19 %. More micro-batches means smaller ones, and a batch has a size limit.

Llama 3 uses an interleaved schedule, with several stages (V) on each GPU, which brings the bubble down to (p − 1)/(V·m) of the computing time. The paper does not give its V and m.

*idle share (p − 1)/(m + p − 1) · 16 stages: 48 % at m = 16, 19 % at 64, 10.5 % at 128*

## 10 Four ways at once

Llama 3 stacks these splits together; the paper calls it 4D parallelism: tensor (TP), context (CP), pipeline (PP) and data (DP). 16,384 GPUs are 8 × 1 × 16 × 128. At the end of training, for sequences of 131,072 tokens, context parallelism goes to 16 ways, cutting each sequence into pieces on different GPUs, and data parallelism drops to 8 copies: 8 × 16 × 16 × 8, still 16,384.

They nest in the order \[TP, CP, PP, DP\]: the further in, the more traffic and the less waiting it can bear, so the closer together it is placed. Tensor parallelism stays inside a server; data parallelism is outermost, across several levels of network, because it can prefetch weights and reduce gradients while computing, so it can wait.

Mixture-of-experts models add a fifth split, putting different experts on different GPUs: see the plate “The Mixture of Experts”. Llama 3 is a dense model and does not use it.

*TP 8 × CP 1 × PP 16 × DP 128 = 16,384 · long sequences: 8 × 16 × 16 × 8*

## 11 Two-fifths of the peak

How well the hardware is used is often given as model FLOPs utilisation (MFU): the model’s computing actually done per second, divided by the GPUs’ peak. On 8,192 GPUs, Llama 3 did about 430 trillion operations a second per GPU, 43 % of the peak; on 16,384, 400 trillion, 41 %; on long sequences, 380 trillion, 38 %.

The rest goes on waiting for communication, the pipeline’s bubbles, and operations that are limited by memory bandwidth and cannot fill the arithmetic units. From 8,192 to 16,384 GPUs utilisation fell from 43 to 41 %: the paper says this is because the batch stayed the same size, so each copy got half as much data.

NVIDIA gives the H100’s peak as 1,979 trillion BF16 operations a second, but that figure assumes sparse matrices; for dense ones it is half, about 989 trillion. The paper does not say which peak it uses; its numbers fit about 989 trillion.

*43 % (8,192 GPUs) · 41 % (16,384) · 38 % (long sequences) · peak about 989 trillion a second*

## 12 466 stops in 54 days

Training is synchronous: at every step all the GPUs exchange data, and one GPU failing can mean restarting the whole job. The paper looked at 54 days of pre-training: the job was interrupted 466 times, 47 of them planned (firmware upgrades, configuration changes) and 419 unexpected.

Of the unexpected ones, faulty GPUs caused 148, the GPUs’ HBM3 memory 72, software bugs 54, network switches and cables 35, unplanned host maintenance 32. The paper puts about 78 % down to hardware, confirmed or suspected. Only three needed significant manual work; automation handled the rest, and effective training time stayed above 90 %.

The remedy is checkpoints: every so often each GPU saves its share of the model’s state, from 1 MB to 4 GB; after a failure the broken machine is swapped out and training resumes from the last checkpoint. The time of day shows too: at hotter middays the GPUs slowed their clocks, and throughput varied by 1⁠–⁠2 %. When all the GPUs pause at once for a checkpoint or for communication, the building’s power draw can swing by tens of megawatts in an instant.

*466 interruptions · 419 unexpected · about 78 % hardware · 3 needed manual work · over 90 % effective training time*

## 13 Taken apart

The GPU tray lifts piece by piece: the gold-topped heat sinks, the eight GPUs, the four NVSwitches with their heat sink, the HGX board, leaving the tray itself.

Rest the pointer on a piece (tap it on a phone) and it lifts on its own, named, while the rest fade.

## 14 Laboratory

Two sliders: the number of pipeline stages p (1 to 16), and the number of micro-batches m (1 to 64) the batch is cut into. From the cut-open end, the first p servers on the plinth each run a stage: blue is the forward pass, pale blue the backward pass, dark is waiting.

The readouts are each GPU’s idle share, (p − 1)/(m + p − 1), and how many times faster it is than one stage alone, pm/(m + p − 1). This is the page’s model of a GPipe-style schedule: the backward pass counts as twice the forward, communication takes no time, and smaller micro-batches are assumed to use the GPU as well as large ones.

**Try this**: In the lab, set 16 stages and 16 micro-batches: how much of the time is each GPU idle? And with 64 micro-batches?

## What the model leaves out

A row of eight racks and a network rack is the page’s choice: Meta has not published how its halls are laid out, nor whether the 128 GPUs of one copy of the model stand together. The servers follow the Grand Teton specification’s sizes (380 × 537 × 805 mm) and published renders; the faces, the parts in the trays and the fan wall are drawn as a likeness. The GPU tray follows NVIDIA’s HGX specification (a 553 × 416 mm board) and the top view in the DGX H100 user guide: eight GPUs four across and two deep, the NVSwitches in front. The SXM module’s size is not published and is estimated from photographs. The 600 mm wide, 1,070 mm deep racks are a common Open Rack V3 frame; the standard fixes only the inside. The power shelf and empty spaces in each rack follow common practice; the network rack holds two modular switches, while Meta says only that the network is “based on the Arista 7800 and Minipack2”. The yellow trough overhead is a usual way to run fibre, not something Meta published. The moving colours are schematic: orange for tensor parallelism, blue for pipeline stages at work, green for data-parallel gradients, all slowed down a great deal.

## Easy to get wrong

Parallelism does not make a model smarter; it gets the same amount of computing done sooner. A utilisation of 41 % does not mean more than half was wasted: at this scale, waiting for communication and the pipeline’s bubbles are hard to avoid entirely. In a ring all-reduce the data each GPU sends barely grows with the number of GPUs; the steps and the delay do. NVLink’s 900 GB/s counts both directions; against the network compare one direction, 450 against 50, which is 9 times; 400 Gb/s is bits, 50 GB/s is bytes. The H100’s 1,979 trillion assumes sparse matrices. Llama 3 used up to 16,384 GPUs, not all 24,576 of the cluster, and it was the Ethernet (RoCE) cluster, not the InfiniBand one. The Llama 3 cluster is air-cooled, not water-cooled. One row of the percentages in the paper’s Table 5 does not match its count (148 interruptions, printed as 30.1 %), so this page quotes the counts and the paper’s own “about 78 %”.

## Common questions

### How many GPUs trained Llama 3 405B?

To train the largest model of its Llama 3 family, Meta used up to 16,384 NVIDIA H100 GPUs at once. The model has 405 billion parameters; it read 15.6 trillion tokens and took about 3.8 × 10²⁵ floating-point operations.

See: [01 128 copies of 128](https://world.lapla.si/en/cluster/#scale)

### Why can’t one GPU hold a large model?

One H100 holds 80 GB in the HBM3 memory stacks on either side of its chip. The 405 billion parameters alone, at 2 bytes each, take 810 GB: more than ten GPUs’ worth.

See: [02 Too much for one](https://world.lapla.si/en/cluster/#alone)

### How much data does each GPU send in a ring all-reduce?

Each GPU sends 2(N − 1)/N times the gradient’s size: 1.75 times for 8 GPUs, 1.98 times for 128, so it barely grows with N.

See: [06 Round the ring](https://world.lapla.si/en/cluster/#ring)

### What is the bubble in pipeline parallelism?

At the start of the pipeline the later stages wait for the first micro-batch; at the end the earlier stages wait for the last gradients. That idle time is called the bubble.

See: [09 The bubble](https://world.lapla.si/en/cluster/#bubble)

### How often was Llama 3’s training interrupted?

The paper looked at 54 days of pre-training: the job was interrupted 466 times, 47 of them planned (firmware upgrades, configuration changes) and 419 unexpected.

See: [12 466 stops in 54 days](https://world.lapla.si/en/cluster/#faults)

## Sources

1. [Llama Team, AI @ Meta (2024): The Llama 3 Herd of Models, arXiv:2407.21783 (up to 16,384 H100s; 405 billion parameters; 15.6 trillion tokens; 3.8 × 10²⁵ operations; the network; 4D parallelism; Table 4’s layouts and utilisation; Table 5’s 466 interruptions)](https://arxiv.org/abs/2407.21783)
2. [Shoeybi et al. (2019): Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism, arXiv:1909.08053 (column and row splits; two all-reduces forward and two backward per layer)](https://arxiv.org/abs/1909.08053)
3. [Narayanan et al. (2021): Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM, SC21, arXiv:2104.04473 (tensor parallelism within a server; bubble (p − 1)/m; the interleaved schedule)](https://arxiv.org/abs/2104.04473)
4. [Huang et al. (2019): GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism, NeurIPS 2019, arXiv:1811.06965 (micro-batches; bubble O((K − 1)/(M + K − 1)); negligible when M ≥ 4K)](https://arxiv.org/abs/1811.06965)
5. [Rajbhandari et al. (2020): ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, SC20, arXiv:1910.02054 (16 bytes a parameter for mixed-precision Adam; the three sharding stages)](https://arxiv.org/abs/1910.02054)
6. [Patarasuk & Yuan (2009): Bandwidth optimal all-reduce algorithms for clusters of workstations, J. Parallel Distrib. Comput. 69(2), 117 (the 2(N − 1)/N lower bound; the ring algorithm)](https://doi.org/10.1016/j.jpdc.2008.09.002)
7. [NVIDIA: nccl-tests, PERFORMANCE.md (all-reduce bus bandwidth = algorithm bandwidth × 2(n − 1)/n)](https://github.com/NVIDIA/nccl-tests/blob/master/doc/PERFORMANCE.md)
8. [NVIDIA: H100 product page (80 GB HBM3; 3.35 TB/s; BF16 1,979 trillion with sparsity; up to 700 W)](https://www.nvidia.com/en-us/data-center/h100/)
9. [NVIDIA (2022): NVIDIA H100 Tensor Core GPU Architecture whitepaper (18 fourth-generation NVLink links of 25 GB/s each way; the third-generation NVSwitch)](https://resources.nvidia.com/en-us-tensor-core/gtc22-whitepaper-hopper)
10. [NVIDIA (2022): Introducing NVIDIA HGX H100 (eight H100s and four NVSwitches, every GPU wired to all four)](https://developer.nvidia.com/blog/introducing-nvidia-hgx-h100-an-accelerated-server-platform-for-ai-and-high-performance-computing/)
11. [NVIDIA: DGX H100/H200 User Guide (the GPU tray from above; air-cooled, front to back)](https://docs.nvidia.com/dgx/dgxh100-user-guide/dgxh100-user-guide.pdf)
12. [NVIDIA (2022): HGX Form Factor Specification, OCP (baseboard 553.0 × 416.0 × 153.2 mm with heat sinks)](https://www.opencompute.org/documents/open-compute-specification-hgx-baseboard-contribution-r1-v0-1-pdf)
13. [Meta (2023): Grand Teton Platform Specification v1.0, OCP (8 OU; 380 × 537 × 805 mm; 118.7 kg; three trays; eight 400G cards; 16 fans; all air-cooled)](https://www.opencompute.org/documents/grand-teton-platform-specification-v1-0-pdf/)
14. [Meta (2021): Minipack2 Switch System Specification, OCP (4 RU; 64 400G ports)](https://www.opencompute.org/documents/minipack2-generic-ocp-specification-rev1-0-pdf/)
15. [OCP (2022): Open Rack Base Frame V3 Specification (the 48 mm OpenU; no outer size set)](https://www.opencompute.org/documents/open-rack-base-specification-version-3-pdf)
16. [Meta Engineering (2024): Building Meta’s GenAI Infrastructure (two clusters of 24,576 GPUs)](https://engineering.fb.com/2024/03/12/data-center-engineering/building-metas-genai-infrastructure/)
17. [Meta Engineering (2024): How Meta trains large language models at scale (700 W; air cooling only; spare capacity)](https://engineering.fb.com/2024/06/12/data-infrastructure/training-large-language-models-at-scale-meta/)
18. [Gangidi et al. (2024): RDMA over Ethernet for Distributed AI Training at Meta Scale, SIGCOMM 2024 (copper inside the rack; single-mode fibre and 400G transceivers to the cluster switches)](https://engineering.fb.com/wp-content/uploads/2024/08/sigcomm24-final246.pdf)

## Data

The row follows what Meta published about the clusters that trained Llama 3 (Llama Team, AI @ Meta, arXiv:2407.21783; Meta Engineering, 2024) and the Grand Teton, Minipack2 and Open Rack V3 specifications (Open Compute Project); the GPU tray follows NVIDIA’s HGX H100 specification and the DGX H100 user guide. Published sizes are used where they exist; the rest is drawn from photographs, and the notes list what is a modelling choice. The memory, time, all-reduce and pipeline figures and the laboratory’s schedule are computed by this page (src/topics/cluster/science.js, tests/cluster.test.js); the schedule is a model, not Meta’s. Material presets, parts and the softbox studio of the site’s model kit are ported from AirsupHQ/machine-skill (MIT, © 2026 Air Supply Technologies Inc.).

## Licence CC BY-NC 4.0

The text, pictures and models on this page are licensed CC BY-NC 4.0: free to use and adapt for teaching and other non-commercial purposes, with credit to world.lapla.si. Third-party material listed above keeps its own licence.

https://creativecommons.org/licenses/by-nc/4.0/deed.en

## Cite this plate

Laplace’s Mini World, Plate F68, “16,384 GPUs, One Model”, https://world.lapla.si/en/cluster/, CC BY-NC 4.0, updated 2026-11-05.