2. Calculus and matrix calculus

Calculus answers one question: if I nudge this, how much does that change? Training a model means turning billions of knobs, and calculus tells you which way to turn each one.

Why it exists

A loss function measures how wrong a model is. To improve it you need, for each parameter, the sensitivity of the loss to that parameter. Guessing is hopeless with billions of parameters, so you need a way to compute all sensitivities exactly and cheaply.

Core ideas

1. Limits: making “infinitely close” precise

A limit describes what a function approaches as its input approaches a point, without ever needing to reach it. It lets you talk about the slope at a single point, where “rise over run” would divide by zero.

You only need the intuition plus the definition of the derivative. The ε–δ machinery is worth seeing once, in Pillar 0’s fluency check, but rarely needed day to day.

2. The derivative is the best linear approximation

The textbook says the derivative is a slope. The first-principles view is stronger: near a point, every smooth function looks like a straight line, and the derivative is that line’s slope.

This view is what generalizes. In many dimensions the “best linear approximation” becomes a vector (the gradient) or a matrix (the Jacobian). Hold on to this sentence and the rest of the pillar follows.

3. The rules, and the one that matters most

Memorize a small table: (xⁿ)’ = n·xⁿ⁻¹, (eˣ)’ = eˣ, (ln x)’ = 1/x, plus the sum, product and quotient rules. Activation functions add a few more: the sigmoid’s derivative is σ(x)(1 − σ(x)), and ReLU’s is 0 for negative inputs and 1 for positive ones.

The chain rule is the one that runs all of deep learning. If y depends on u and u depends on x, sensitivities multiply along the chain:

Intuition: if u moves 3 times as fast as x, and y moves 2 times as fast as u, then y moves 6 times as fast as x. A deep network is a long chain, so its gradient is a long product.

4. Partial derivatives

When a function has many inputs, a partial derivative ∂f/∂xᵢ asks how f changes when you nudge only xᵢ and freeze everything else. It is an ordinary derivative with the other variables treated as constants.

5. The gradient: direction of steepest ascent

Collect all partial derivatives into one vector and you get the gradient ∇f. It points in the direction where f increases fastest, and its length says how steep that increase is.

Why steepest? The rate of change in a unit direction u is the dot product ∇f · u. From Pillar 1, a dot product is largest when the two vectors point the same way. So the steepest direction is ∇f itself, and the steepest descent is −∇f.

The gradient is also perpendicular to the level sets (contour lines) of f. Picture a hiking map: the gradient crosses contour lines at right angles, straight uphill.

6. The Jacobian: the derivative of a vector function

For a function from ℝⁿ to ℝᵐ, the best linear approximation is a matrix: the Jacobian J, with Jᵢⱼ = ∂fᵢ/∂xⱼ. Its shape is m×n, matching the function’s output and input sizes.

Now the chain rule in many dimensions becomes beautiful. Composing functions means multiplying their Jacobians. Composition of functions on one side, matrix multiplication on the other: Pillar 1 and Pillar 2 are the same idea.

7. The Hessian: curvature

The Hessian H is the matrix of second derivatives, Hᵢⱼ = ∂²f/∂xᵢ∂xⱼ. It describes how the gradient itself changes: the curvature of the landscape. For smooth functions it is symmetric.

At a point where the gradient is zero, the Hessian’s eigenvalues tell you what you found. All positive means a bowl (local minimum), all negative means a hill top, and mixed signs mean a saddle. In high dimensions, saddles vastly outnumber minima.

8. Taylor expansion: the local picture, with curvature

Taylor’s formula adds terms to the linear approximation. Up to second order in many dimensions:

Nearly all optimization theory starts here. Gradient descent trusts only the first-order term, Newton’s method uses the second-order term, and learning-rate limits come from how badly the first-order picture ignores curvature.

9. Backpropagation and automatic differentiation

A model is a computational graph: inputs flow through simple operations to a single scalar loss. The forward pass computes every intermediate value. The backward pass applies the chain rule from the loss back to every parameter, reusing results along the way.

The crucial fact: reverse-mode differentiation gives the gradient for all parameters at a cost of only a small constant multiple of one forward pass. That holds whether you have ten parameters or a trillion. It is the reason training huge models is possible at all.

Reverse mode wins because a loss has many inputs and one output. Forward mode is the opposite and wins when there are few inputs and many outputs. Libraries never build full Jacobians; they compute vector–Jacobian products, which is far cheaper.

The price is memory: the backward pass needs the intermediate values from the forward pass. That is why activation memory dominates training and why tricks like activation checkpointing (recompute instead of store) exist.

10. Matrix calculus you will actually use

The shape rule saves you here: the gradient of a scalar loss with respect to a matrix has the same shape as that matrix. With that, a few identities cover most derivations.

Expression	Gradient	Note
aᵀx with respect to x	a	the linear case
xᵀAx with respect to x	(A + Aᵀ)x	equals 2Ax when A is symmetric
‖x‖² with respect to x	2x	behind weight decay
y = Wx: loss gradient for W	(∂L/∂y) xᵀ	an outer product, same shape as W
y = Wx: loss gradient for x	Wᵀ (∂L/∂y)	gradients flow back through the transpose

The last two lines are backprop through a linear layer, the most common operation in deep learning.

11. Integrals: accumulation

An integral adds up infinitely many infinitely small pieces: area, total change, accumulated probability. The fundamental theorem of calculus says integration and differentiation undo each other.

In AI you meet integrals mostly as expectations over continuous variables and as normalizing constants. They are usually impossible to compute exactly. The universal trick is Monte Carlo: replace the integral with an average over samples, which is exactly what a minibatch does.

In AI

The most important gradient in deep learning is softmax followed by cross-entropy. With logits z, predicted probabilities p = softmax(z), and a one-hot target y, it collapses to:

Read it: push up the correct class by how much probability it is missing, and push down each wrong class by how much it wrongly received. Derive it once yourself and you will never forget it.

Calculus also explains several architecture choices. A sigmoid’s derivative is at most 0.25, so a product of many shrinks toward zero: vanishing gradients. ReLU’s derivative is exactly 1 on its active side, which helps.

Residual connections compute x + f(x), whose derivative is 1 + f’(x). That built-in “1” gives gradients a clean highway back through dozens of layers.

Some operations have no useful derivative: argmax, sampling, rounding. That is why discrete choices need special tricks, such as straight-through estimators, Gumbel-softmax, or the REINFORCE estimator used in reinforcement learning.

Questions to ask

• What exactly is being differentiated, with respect to what?
• What is the shape of this gradient, and does it match the parameter’s shape?
• Is this function differentiable everywhere it is used? What happens at the kinks?
• Along this chain of layers, which factors could make the product vanish or explode?
• What must be stored from the forward pass to compute this backward pass, and how much memory is that?
• What does the first-order approximation miss here, and does curvature matter?
• If a gradient looks wrong, how do I check it numerically?

Fluency check

• Derive the sigmoid derivative σ(x)(1 − σ(x)) from the definition σ(x) = 1/(1 + e⁻ˣ).
• Derive the softmax–cross-entropy gradient p − y on paper.
• Backpropagate by hand through a two-layer network with scalar weights and a squared-error loss.
• Check any gradient numerically with the central difference (f(x + h) − f(x − h)) / 2h.
• Write a scalar autograd engine in about 100 lines of Python: values, operations, and a backward() that walks the graph in reverse.
• For y = Wx, derive ∂L/∂W and ∂L/∂x and confirm the shapes.