4. Statistics and evaluation

Probability goes from a known distribution to data. Statistics runs the other way: from finite data back to conclusions about the world. Training a model is estimation, and evaluating one is inference.

Why it exists

You never see the true distribution, only a sample of it. Statistics tells you what that sample does and does not justify. In AI systems work, this is the difference between a real improvement and noise that happened to look good.

Core ideas

1. Populations, samples and estimators

The population is everything you care about, like all future user queries. The sample is what you actually have. An estimator is a recipe that turns the sample into a guess about the population, such as the sample mean for the true mean.

Estimators have two kinds of error. Bias is being wrong on average. Variance is being wobbly from sample to sample. A good estimator keeps both small.

2. Maximum likelihood: where loss functions come from

Maximum likelihood estimation picks the parameters under which the observed data is most probable. With i.i.d. data the likelihood is a product, so you maximize its log, which turns the product into a sum:

The simplest example: after 7 heads in 10 flips, the MLE for the coin’s bias is 0.7. Derive it yourself by setting the derivative of the log-likelihood to zero.

Now the first-principles payoff. Training a classifier or language model with cross-entropy loss is exactly maximum likelihood. Training a regressor with mean squared error is maximum likelihood under Gaussian noise, because the Gaussian’s negative log-density is a squared error plus a constant:

So loss functions are not arbitrary choices. Each one is an assumption about how your data is noisy. If that assumption is badly wrong, for example heavy-tailed outliers, the loss is wrong too.

3. Priors, MAP and regularization

Maximum a posteriori (MAP) estimation adds a log prior to the log-likelihood: “what is plausible before seeing data”. A Gaussian prior on the weights adds a penalty proportional to ‖θ‖², which is exactly L2 regularization, or weight decay.

A Laplace prior gives an L1 penalty, which pushes weights to exactly zero. Full Bayesian inference keeps the whole posterior instead of one best point; ensembles of models are a practical approximation of that idea.

4. Empirical risk, generalization and overfitting

What you want to minimize is the true risk: the expected loss over the real data distribution. What you can minimize is the empirical risk: the average loss on your training set. Generalization is the gap between the two.

Overfitting means memorizing the sample’s quirks, so training loss falls while test loss rises. Underfitting means the model cannot capture the real pattern at all. That is why you split data into training, validation and test sets, and touch the test set only once.

For squared error, expected test error splits cleanly into bias², variance and irreducible noise. The classical lesson is a trade-off: more flexible models trade bias for variance.

Modern deep learning complicates that story. Hugely over-parameterized networks often generalize well, and test error can fall again past the point of perfect fit (“double descent”). Know the classical picture, and know it is not the whole truth.

5. Uncertainty on a number: standard errors and intervals

Every metric computed on a sample is itself a random variable. For an accuracy p measured on n independent questions, the standard error is:

A rough 95% confidence interval is p ± 2·SE. At 80% accuracy on 100 questions, SE is 4 points, so the interval is roughly 72% to 88%. On 1,000 questions it narrows to about ±2.5 points.

So a 2-point gain on a 100-question benchmark is almost certainly noise. This one calculation will save you from a large share of bad decisions in AI systems work.

6. Hypothesis tests, p-values and the bootstrap

A hypothesis test asks whether data is surprising under a null hypothesis, such as “both models are equally good”. The p-value is the probability of data at least this extreme if the null were true.

The p-value is not the probability that the null is true. It is also not the size of the effect. Misreading it is the most common statistical error in published work.

Two practical upgrades matter most. Paired comparisons score both models on the same questions and analyse the per-question differences, which removes question difficulty from the noise. The bootstrap resamples your data with replacement thousands of times to get an interval for any metric, no formula needed.

Beware multiple comparisons. Try 20 prompt variants and one will “win” at p < 0.05 by luck alone. The more you search, the more your best result is inflated.

7. Metrics and what they hide

Metric	Definition	Use it when	Watch out for
Accuracy	correct ÷ total	classes are balanced	misleading for rare events
Precision	TP ÷ (TP + FP)	false alarms are costly	ignores what you missed
Recall	TP ÷ (TP + FN)	misses are costly	ignores false alarms
F1	harmonic mean of precision and recall	you need one number for both	hides which one is weak
ROC AUC	ranking quality across all thresholds	scores will be thresholded later	can look good on very imbalanced data
Calibration error	gap between confidence and accuracy	probabilities drive decisions	low accuracy can still be calibrated
pass@k	chance at least one of k samples is correct	code and reasoning tasks	rewards breadth, not reliability
Win rate (LLM judge)	share of pairwise wins	open-ended generation	judge bias toward length, position, style

TP, FP and FN are true positives, false positives and false negatives from the confusion matrix. Goodhart’s law applies to all of them: once a metric becomes a target, it stops measuring what you meant.

8. Correlation, causation and experiments

Correlation can come from a hidden confounder that drives both variables. Simpson’s paradox shows a trend can even reverse when you split data into groups.

Randomized experiments break confounding, because chance decides who gets each treatment. That is why A/B tests, with random assignment of users to model or prompt variants, are the gold standard for product decisions.

9. Distribution shift and contamination

Covariate shift means inputs change between training and deployment; label shift means the balance of outcomes changes. Either breaks the i.i.d. assumption behind every number above.

Contamination means test questions leaked into training data. Then a benchmark measures memory, not ability. For public benchmarks and web-scale training data, assume this risk is real.

In AI

• Pre-training an LLM is maximum likelihood: next-token cross-entropy is the negative log-likelihood of the training text.
• Weight decay is a Gaussian prior: MAP estimation in disguise.
• Evals are statistics: report intervals, compare models paired on the same items, and run multiple seeds.
• LLM-as-judge is a measurement instrument: check it against human labels before you trust it.
• Production monitoring is shift detection: compare live input distributions against what the model was built for.

Questions to ask

• What population do I care about, and is my sample representative of it?
• What is the uncertainty on this number, and how many examples produced it?
• Is the difference larger than the noise? Was the comparison paired?
• How many variants were tried before this result was reported?
• Could the test data have leaked into training?
• What noise assumption does this loss function encode?
• Does this metric measure what I actually care about, or a convenient proxy?

Fluency check

• Derive the maximum likelihood estimate for a coin from n flips.
• Show that minimizing squared error equals maximizing a Gaussian likelihood.
• Compute 95% intervals for 80% accuracy on 100 and on 1,000 questions.
• Bootstrap an interval for a model’s accuracy in NumPy.
• Simulate two identical models on a 100-question benchmark and count how often one “wins” by 3 or more points.
• Build a confusion matrix from predictions and compute precision, recall and F1 by hand.