18. Fluency kit

Fluency means you can explain each idea from first principles, estimate its costs on the back of an envelope, and ask the question that exposes a weak claim. This section collects the vocabulary, the checklists, the core papers and a build path to get there.

Glossary

Term	Meaning
Token	the unit a model reads and writes; usually a subword
Embedding	a learned vector for a token, or for a whole text in retrieval
Logits	raw scores over the vocabulary before softmax
Softmax	turns scores into a probability distribution
Cross-entropy	−log of the probability given to the correct token; the training loss
Perplexity	exp(cross-entropy); the effective number of choices
Residual stream	the per-token vector every block reads from and adds to
Q, K, V	query, key and value projections used by attention
Attention head	one attention operation in its own subspace
Causal mask	stops tokens attending to future positions
MLP / FFN	the per-token two-layer network inside each block
SwiGLU	gated MLP variant used in most modern LLMs
LayerNorm / RMSNorm	per-token normalization; RMSNorm skips the mean
Pre-norm	normalization applied to sublayer inputs, not the residual path
RoPE	rotary position embedding: relative position via rotating Q and K
Context window	maximum number of tokens processed at once
KV cache	stored keys and values of past tokens, reused during generation
Prefill / decode	parallel prompt processing / token-by-token generation
TTFT / TPOT	time to first token / time per output token
MQA / GQA	query heads sharing one / a few key–value heads to shrink the KV cache
FlashAttention	exact attention kernel that never stores the T × T matrix
Quantization	storing weights or activations in fewer bits
Speculative decoding	a draft model proposes tokens; the target model verifies them in parallel
MoE	mixture of experts: only some MLP experts run per token
Active parameters	parameters actually used per token in an MoE
SSM / Mamba	linear-recurrence sequence layer with constant inference memory
Scaling laws	power laws linking loss to model size, data and compute
Chinchilla-optimal	compute-optimal ratio, about 20 tokens per parameter
MFU	share of peak hardware FLOPs actually used
ZeRO / FSDP	sharding optimizer state, gradients and weights across GPUs
SFT	supervised fine-tuning on demonstrations
RLHF	reinforcement learning from human feedback via a reward model
DPO	preference optimization directly on pairs, no reward model
RLVR / GRPO	RL with verifiable rewards / a group-relative policy-gradient algorithm
Chat template	token format marking roles and turns in a conversation
In-context learning	learning a task from examples in the prompt, without weight updates
LoRA / QLoRA	low-rank adapters / LoRA on a 4-bit base model
Distillation	training a small student to imitate a larger teacher
Bi-encoder / cross-encoder	separate encoding for fast retrieval / joint encoding for accurate reranking
ANN / HNSW	approximate nearest-neighbour search / a graph-based ANN index
RAG	retrieval-augmented generation
Induction head	attention circuit that completes repeated patterns
Superposition	many features packed into fewer dimensions
Prompt injection	untrusted text that steers a model against its instructions
Constrained decoding	masking tokens so output follows a grammar or schema

Questions to ask about any model

1. Architecture: dense or MoE (total and active parameters)? L, d, heads, KV heads? Attention variants (sliding window, hybrid)?
2. Position and context: which encoding and base? Trained context vs advertised? Evidence it uses long context?
3. Tokenizer: vocabulary size; tokens per word in your languages?
4. Data and compute: tokens trained, data mix, tokens per parameter?
5. Post-training: SFT, preference optimization, RL with verifiers? A reasoning mode?
6. Evaluation: which benchmarks and prompts, any contamination checks, any results on tasks like yours?
7. Inference: KV cache per token, memory at your context length, quantized variants and their quality?
8. Deployment: licence, API vs weights, data handling, version stability.

Questions to ask about any paper or claim

1. What problem, and what was the strongest baseline?
2. Is the comparison at equal compute, parameters and data?
3. Which ablations show the new component is the reason for the gain?
4. At what scale was it tested, and does it hold as scale grows?
5. Are the gains larger than run-to-run noise?
6. What does it cost at inference?

Core reading list

Read roughly in this order. Titles and years are from memory; find each on arXiv by its title.

#	Paper	Year	Why read it
1	Bahdanau et al., Neural Machine Translation by Jointly Learning to Align and Translate	2014	attention’s origin, attached to RNNs
2	He et al., Deep Residual Learning for Image Recognition	2015	residual connections
3	Vaswani et al., Attention Is All You Need	2017	the transformer
4	Devlin et al., BERT	2018	encoders and masked modeling
5	Radford et al., Language Models are Unsupervised Multitask Learners (GPT-2)	2019	decoder-only scaling, zero-shot tasks
6	Kaplan et al., Scaling Laws for Neural Language Models	2020	power laws
7	Brown et al., Language Models are Few-Shot Learners (GPT-3)	2020	in-context learning at scale
8	Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks	2020	RAG
9	Dosovitskiy et al., An Image is Worth 16x16 Words (ViT)	2020	transformers for vision
10	Radford et al., Learning Transferable Visual Models From Natural Language Supervision (CLIP)	2021	contrastive multimodal learning
11	Su et al., RoFormer (RoPE)	2021	rotary positions
12	Hu et al., LoRA	2021	low-rank adaptation
13	Elhage et al., A Mathematical Framework for Transformer Circuits	2021	residual stream, QK/OV circuits
14	Hoffmann et al., Training Compute-Optimal Large Language Models (Chinchilla)	2022	compute-optimal scaling
15	Dao et al., FlashAttention	2022	IO-aware exact attention
16	Ouyang et al., Training language models to follow instructions with human feedback (InstructGPT)	2022	SFT and RLHF
17	Olsson et al., In-context Learning and Induction Heads	2022	a mechanism for in-context learning
18	Leviathan et al., Fast Inference from Transformers via Speculative Decoding	2022	speculative decoding
19	Bai et al., Constitutional AI	2022	AI feedback from written principles
20	Rafailov et al., Direct Preference Optimization	2023	preference tuning without RL
21	Dettmers et al., QLoRA	2023	4-bit fine-tuning
22	Ainslie et al., GQA	2023	grouped-query attention
23	Kwon et al., Efficient Memory Management for LLM Serving with PagedAttention (vLLM)	2023	KV-cache memory management
24	Gu & Dao, Mamba	2023	selective state space models
25	Jiang et al., Mixtral of Experts	2024	sparse MoE in practice
26	DeepSeek-AI, DeepSeek-R1	2025	RL with verifiable rewards for reasoning

Alongside the papers:

• Andrej Karpathy’s “Neural Networks: Zero to Hero” videos and the nanoGPT repository: build everything from scratch.
• Jay Alammar’s “The Illustrated Transformer”: visual intuition for Parts 4–5.
• “The Annotated Transformer” (Harvard NLP): the 2017 paper as runnable code.
• Stanford CS336, Language Modeling from Scratch: a systems-heavy course matching Parts 8–10.
• Sebastian Raschka, “Build a Large Language Model (From Scratch)”: a book-length build.
• Anthropic’s Transformer Circuits thread: interpretability research behind Part 15.
• 3Blue1Brown’s neural network and transformer videos: geometric intuition.

Learning path with build projects

Each project makes you implement part of this guide. Tick them off as you go.

• 1. Autograd from scratch (Parts 1–2). Build a tiny scalar autograd engine, micrograd-style, and train an MLP. Then reproduce one run in PyTorch and match the gradients.
• 2. Tokenizer (Part 3). Implement BPE training and encoding; compare your token counts with a production tokenizer on English, German and code.
• 3. Attention by hand (Part 4). Implement scaled dot-product attention with a causal mask and test it against F.scaled_dot_product_attention. Remove the √d_k scaling and watch training degrade.
• 4. GPT from scratch (Parts 5–7). Build a small decoder-only model, nanoGPT-style, on a text corpus. Count parameters by formula and by code. Ablate residuals, post-norm vs pre-norm, and learned positions vs RoPE.
• 5. Scaling experiment (Part 8). Train four or five model sizes on the same data; plot loss against compute on log–log axes and fit a power law.
• 6. Inference engine (Part 9). Add a KV cache and sampling (temperature, top-p) to your model; measure tokens/s with and without the cache. Run a quantized open model and compare speed and quality.
• 7. Fine-tune (Parts 11–12). LoRA-fine-tune a small open model on a narrow task, then run DPO on preference pairs. Compare both against a prompting baseline on a held-out set.
• 8. RAG with evals (Part 13). Build hybrid retrieval with reranking over documents you know well; label 50 questions; report recall@k and faithfulness; iterate on chunking.
• 9. Look inside (Part 15). Use TransformerLens on a small model: logit lens, attention patterns on repeated text, and find an induction head by ablation.
• 10. Agent with defences (Parts 16–17). Build a tool-using agent and write trajectory evals. Attack it with indirect prompt injection, add least-privilege and confirmation gates, and measure cost and latency per task.

You’re fluent when you can…

• Derive scaled dot-product attention and explain every piece without notes.
• Estimate parameters, training FLOPs, training memory and KV cache size from a model config.
• Explain why decode is memory-bound, and name three optimizations that follow from it.
• Explain how a base model becomes an assistant, and where sycophancy and verbosity come from.
• Choose between prompting, RAG, LoRA and distillation for a concrete problem, and defend the choice with costs.
• Read a new architecture paper and say what problem it solves, what it trades away and which experiment would test it.