Transformers from First PrinciplesHow to use this guide · 18 of 18 written
  1. How to use this guide
  2. 01The machinery: how a neural network computes and learns
  3. 02Making deep networks trainable
  4. 03Turning text into numbers
  5. 04Attention from first principles
  6. 05The Transformer block
  7. 06Position: telling the model where each token is
  8. 07Architecture families and training objectives
  9. 08Pretraining at scale
  10. 09Inference and serving
  11. 10Efficiency variants: MoE, long context and beyond attention
  12. 11Post-training: from text predictor to assistant
  13. 12Adapting models: prompting, fine-tuning, LoRA and distillation
  14. 13Embeddings, retrieval and RAG
  15. 14Beyond text: vision, speech and multimodal models
  16. 15Inside the model: interpretability
  17. 16Evaluation, failure modes and security
  18. 17From model to system
  19. 18Fluency kit

Transformers from First Principles

AI Models · 18 posts1h 19m read

A transformer is one idea - each token builds its representation by looking up information from other tokens, weighted by learned relevance - wrapped in standard deep-learning machinery that makes it trainable at huge scale. This guide builds that idea from the ground up: first the machinery, then attention and the block, then how models are trained, served, adapted and turned into systems.

How to use this guide

Every part follows the same arc: the problem, the mechanism (with tensor shapes), the cost, what breaks it, and closing questions. Those questions are the real test of fluency. If one stumps you, reread the part or work it out in code.

The map

Parts Theme What you gain for AI-systems work
1–3 Foundations Read any training loop; reason about loss, gradients, optimizers and tokens
4–7 The architecture Explain attention, the block, positions and model families from scratch
8–10 Scale Estimate compute, memory, latency and cost; understand serving engines
11–13 From model to product Know how assistants are made; choose between prompting, RAG and fine-tuning
14–16 Breadth and depth Multimodal models, interpretability, evaluation and security
17–18 Systems and fluency Tools, agents and cost math; glossary, checklists, reading list, projects

Five questions to ask of any component

  1. What fails without it? Every piece exists because something broke. If you can’t name the failure, you don’t understand the piece yet.
  2. What goes in and out, in shapes? Write the tensor shapes. Most confusion dissolves once shapes are explicit.
  3. What does it cost, and how does that grow? Compute (FLOPs), memory and latency as functions of sequence length T, width d, layers L and parameters N.
  4. What does it assume, and when does that break? Example: learned position embeddings assume you never exceed the trained length.
  5. How would I check? An ablation, a measurement, or a unit test against a reference implementation.

Notation used throughout

Symbol Meaning Typical values (approximate)
B batch size: sequences processed together 1 – thousands
T sequence length in tokens 512 – 1M
d model width (d_model): size of each token’s vector 768 – 16,384
h number of attention heads 12 – 128
d_h head dimension, d / h 64 – 128
L number of layers (blocks) 12 – 120+
V vocabulary size 32k – 256k
N number of parameters 100M – 1T+
D number of training tokens 1B – 30T+

Prerequisites

  • Linear algebra: vectors, dot products, matrix multiplication, projections. A matrix multiply is a batch of dot products; that single fact explains most of the transformer.
  • Calculus: derivatives, the chain rule, the gradient as the direction of steepest increase.
  • Probability: distributions, expectation, log-likelihood, softmax.
  • Code: Python and PyTorch tensors. Reading is not enough; build the pieces (the Fluency kit has a project path).