Transformers from First PrinciplesPost 8 of 18 · 18 of 18 written
  1. How to use this guide
  2. 01The machinery: how a neural network computes and learns
  3. 02Making deep networks trainable
  4. 03Turning text into numbers
  5. 04Attention from first principles
  6. 05The Transformer block
  7. 06Position: telling the model where each token is
  8. 07Architecture families and training objectives
  9. 08Pretraining at scale
  10. 09Inference and serving
  11. 10Efficiency variants: MoE, long context and beyond attention
  12. 11Post-training: from text predictor to assistant
  13. 12Adapting models: prompting, fine-tuning, LoRA and distillation
  14. 13Embeddings, retrieval and RAG
  15. 14Beyond text: vision, speech and multimodal models
  16. 15Inside the model: interpretability
  17. 16Evaluation, failure modes and security
  18. 17From model to system
  19. 18Fluency kit

8. Pretraining at scale

Transformers · 8 of 185 min readNov 2026

Pretraining teaches a decoder to predict the next token over trillions of tokens of text and code, and almost all of a model’s knowledge and raw capability comes from this stage. Three tools make it something you can reason about: the compute formula C ≈ 6ND, memory accounting per parameter, and scaling laws that predict loss from model size and data.

The objective

Given a sequence x₁ … x_T, minimize the average negative log-likelihood of each token given its prefix:

L=−1T∑t=1Tlog⁡pθ(xt∣x<t)\mathcal{L} = -\frac{1}{T}\sum_{t=1}^{T} \log p_\theta(x_t \mid x_{<t})

Why such a simple objective produces general capability: to predict the next token of a physics textbook, a court ruling or a Python file, the model must capture grammar, facts, the structure of arguments and how code behaves. Lower loss also means better compression: by Shannon’s theory, a better predictor encodes the data in fewer bits. Predicting text well forces the model to model the processes that generated it.

Data: the biggest lever

A typical pretraining data pipeline:

  1. Collect: web crawls (such as Common Crawl), code, books, scientific papers, licensed sources, and increasingly synthetic data.
  2. Extract and clean: HTML to text, language identification, boilerplate removal.
  3. Filter for quality: heuristics (length, symbol ratios) plus learned quality classifiers.
  4. Deduplicate: exact and near-duplicate removal (e.g., MinHash). Duplicates waste compute and increase memorization.
  5. Remove unwanted content: personal data, toxic content, and benchmark test sets (decontamination).
  6. Mix: set proportions of web, code, math and other domains, often upweighting high-quality data near the end of training (annealing).

Many gains credited to architecture are really data improvements. When comparing models, ask about tokens seen, data mix and filtering before architecture.

Compute: C ≈ 6ND

Training compute in FLOPs is roughly six times parameters times training tokens: 2N for the forward pass and 4N for the backward pass, per token.

C≈6 N DC \approx 6\, N\, D

Example: a 7B-parameter model on 2T tokens needs 6 × 7·10⁹ × 2·10¹² ≈ 8.4·10²² FLOPs. An A100 peaks at about 312 TFLOP/s in BF16; at a realistic 40% utilization that is about 1.25·10¹⁴ FLOP/s, so ≈ 6.7·10⁸ GPU-seconds ≈ 190k GPU-hours. The Llama 2 paper reported roughly 184k A100-hours for its 7B model, so the formula holds up (figures approximate, from memory).

The share of peak hardware FLOPs actually used is called model FLOPs utilization (MFU); roughly 35–55% is typical for well-tuned large runs.

Memory: where the bytes go

Training with mixed-precision AdamW needs about 16 bytes per parameter before activations:

Item Bytes per parameter
BF16 weights 2
BF16 gradients 2
FP32 master weights 4
Adam first moment m (FP32) 4
Adam second moment v (FP32) 4
Total ≈ 16

A 7B model therefore needs about 112 GB for this alone - more than one 80 GB GPU - plus activations, which grow with batch size × sequence length × width × layers. Activation checkpointing saves memory by discarding activations and recomputing them in the backward pass, at the cost of roughly one extra forward pass (about a third more compute).

Inference, by contrast, needs only the weights - 2 bytes per parameter in BF16, so 14 GB for 7B - plus the KV cache (Part 9).

Scaling laws

Loss falls as a smooth power law as you increase parameters N, data D and compute C, across many orders of magnitude. That predictability let labs plan huge runs from small experiments.

The Chinchilla analysis (Hoffmann et al., 2022) fit:

L(N,D)=E+ANα+BDβL(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}

E is the irreducible entropy of text; the other two terms shrink as the model and data grow. Minimizing this for a fixed budget C = 6ND gives the compute-optimal recipe: grow N and D in roughly equal proportion, about 20 training tokens per parameter. Chinchilla (70B parameters, 1.4T tokens) beat the four-times-larger Gopher trained with the same compute on less data.

The practical twist: compute-optimal for training is not optimal for deployment. Inference cost scales with N, so a smaller model trained much longer is cheaper to serve. Modern models are deliberately “over-trained”: Llama 3 8B saw about 15T tokens, roughly 1,900 tokens per parameter. When someone calls a model “Chinchilla-optimal”, ask whether they optimized for training or for inference.

Emergence: a caution

Some abilities seem to switch on suddenly at a certain scale. Part of this is a measurement artifact: all-or-nothing metrics, like exact match on a five-step problem, jump abruptly even when per-token accuracy improves smoothly. Smooth metrics usually show smooth progress, so treat claims of sudden capabilities with care.

Distributed training: splitting the work

No single GPU can hold or train a frontier model. Parallelism strategies are combined:

Strategy What is split Communication Notes
Data parallel (DP) the batch; each GPU holds a full model copy all-reduce of gradients every step simplest; requires the model to fit on one GPU
ZeRO / FSDP optimizer states, gradients and parameters, sharded across DP ranks gather parameters when needed removes redundant copies; the default for mid-size training
Tensor parallel (TP) individual weight matrices within a layer every layer, very frequent needs fast links, usually within one server
Pipeline parallel (PP) groups of layers across GPUs activations between stages micro-batches shrink idle “bubbles”
Sequence / context parallel the sequence dimension attention needs exchange (e.g., ring attention) for very long contexts
Expert parallel (EP) MoE experts across GPUs all-to-all token routing for mixture-of-experts models (Part 10)

Large runs use several at once (“3D parallelism” and beyond). At thousands of GPUs hardware failures are routine, so frequent checkpoints and automatic restarts are part of the training system.

Questions you should now be able to answer

  • Estimate the FLOPs and GPU-hours to train a 70B model on 15T tokens with GPUs delivering 400 TFLOP/s effective.
  • Why does training need about 16 bytes per parameter while inference needs about 2?
  • What does Chinchilla say about a fixed budget, and why do labs deliberately ignore it?
  • Why might an ability look as if it “emerges” suddenly when it doesn’t?
  • Which parallelism do you add first if the model doesn’t fit on one GPU? And if one layer doesn’t fit?