9. Learning path, resources and question bank

Follow the pillars in dependency order, build something in every phase, and finish by building a small GPT. Durations assume about 6–8 focused hours a week; adjust freely.

Learning path

Phase	Pillars	Suggested time	Build this	Done when you can
1	0 Language, 1 Linear algebra	3–4 weeks	SVD image compression; single-head attention in NumPy	explain attention as dot products, projections and a weighted sum
2	2 Calculus	2–3 weeks	a scalar autograd engine with backward()	backpropagate through a small network by hand and by code
3	3 Probability, 4 Statistics	3–4 weeks	a sampler (temperature, top-p) and a bootstrapped eval harness	say whether a benchmark gain is real, with an interval
4	5 Information theory, 6 Optimization	2–3 weeks	SGD, momentum and Adam from scratch; a character-level language model	derive why cross-entropy is the loss and tune a training run
5	7 Numerical and systems	2 weeks	a calculator for weights, KV cache, FLOPs and decode speed	size a GPU and estimate the serving cost for a given model
6	Capstone	3–4 weeks	a tiny GPT from scratch, annotated by pillar	walk through the capstone list without notes

The daily first-principles loop

Each study session should run one concept through the same loop. Skipping steps is how understanding stays shallow.

1. Intuition: watch or read for the picture before the formula.
2. Derive: rebuild the key result on paper with the smallest possible example.
3. Implement: write it in NumPy from scratch, with shapes in comments.
4. Break it: change one assumption (larger values, dependent data, a bigger learning rate) and watch it fail.
5. Explain: write one paragraph in your own words, as if teaching someone.
6. Connect: find the exact line in a real model’s code where it lives.

Resources by pillar

These are standard, widely used references, listed from memory; several are free online from their authors.

Pillar	Intuition first	Depth	Hands-on
0 Language	Daniel Velleman, How to Prove It (first chapters)	Book of Proof by Richard Hammack	write symbol tables for formulas in papers you read
1 Linear algebra	3Blue1Brown, Essence of Linear Algebra	Gilbert Strang, Introduction to Linear Algebra and MIT 18.06 lectures	NumPy: SVD, projections, attention
2 Calculus	3Blue1Brown, Essence of Calculus	Parr and Howard, The Matrix Calculus You Need for Deep Learning	Andrej Karpathy, Neural Networks: Zero to Hero (micrograd)
3 Probability	3Blue1Brown videos on Bayes and the central limit theorem	Blitzstein and Hwang, Introduction to Probability (Harvard Stat 110)	simulate every theorem before proving it
4 Statistics	Wasserman, All of Statistics (selected chapters)	Bishop and Bishop, Deep Learning: Foundations and Concepts	build an eval harness with bootstrap intervals
5 Information theory	Shannon’s 1948 paper, A Mathematical Theory of Communication (first sections)	David MacKay, Information Theory, Inference, and Learning Algorithms	compress text with a simple model
6 Optimization	Goodfellow, Bengio and Courville, Deep Learning, chapters 4 and 8	Boyd and Vandenberghe, Convex Optimization	optimizers from scratch on 2D surfaces
7 Numerical and systems	Goldberg, What Every Computer Scientist Should Know About Floating-Point Arithmetic	How to Scale Your Model (Google DeepMind’s online book); Kaplan et al. 2020 and Hoffmann et al. 2022 on scaling laws	model memory and FLOP calculators
All	Deisenroth, Faisal and Ong, Mathematics for Machine Learning	Vaswani et al. 2017, Attention Is All You Need	Karpathy’s nanoGPT

Master question bank

These are the questions to reach for in each situation. They are how you turn “I don’t get it” into a question with an answer.

When reading a formula

• What does each symbol mean, what set does it live in, and what is its shape?
• Is this a definition or a claim, and what justifies it?
• Which index does each sum or product run over?
• What happens in the extreme cases: zero, one, infinity, a single element?
• Can I compute it by hand for a 2-element example?

When reading a paper

• What problem was impossible or expensive before this?
• What is the baseline, and is the comparison fair: same data, same compute?
• Which assumption makes the method work, and when would it fail?
• How many samples or seeds support each number, and how wide are the intervals?
• What does it cost in FLOPs, memory and latency?

When debugging training

• Is the starting loss right? With uniform predictions over |V| tokens, it should be about ln |V|: around 10.8 for 50,000 tokens.
• Can the model overfit a single small batch? If not, the bug is in the code, not the data.
• What are the gradient norms per layer: vanishing, exploding, or NaN?
• Is a loss spike from the learning rate, a bad batch, or numerical overflow?
• Do the shapes of every gradient match their parameters?

When designing or sizing an AI system

• What dominates memory: weights, KV cache, activations, or optimizer state?
• How many FLOPs and bytes does each token cost, and is it compute-bound or memory-bound?
• What are the latency and throughput targets, and what batch size do they imply?
• What precision and quantization can we afford, measured on our own evals?
• How will we know in production that the input distribution has shifted?

When judging a claim about AI

• What is the base rate, and what is the sample size?
• Was the comparison paired, and how many variants were tried?
• Could the test data have leaked into training?
• Is the metric the real goal, or a proxy that can be gamed?
• Is this correlation, or was it tested with a randomized experiment?

How to ask for help and get a precise answer

The most useful question format states your model of the problem, so the gap becomes visible. Use this template with teachers, colleagues or an AI assistant:

I understand X. I expected Y because Z. Instead I see W. Which part of my reasoning is wrong?

Include the shapes, the smallest failing example, and what you already tried. A question in this form almost always gets a precise answer, and often you find the answer while writing it.