12. Adapting models: prompting, fine-tuning, LoRA and distillation

To make a model good at your task, change its inputs before you change its weights. Prompting and retrieval are cheap, fast to iterate and reversible; fine-tuning is for behaviour prompting can’t reliably produce, or for making a smaller model do a narrow job cheaply. LoRA made fine-tuning affordable by training a tiny low-rank update instead of all the weights.

The decision ladder

1. Prompting: clear instructions, context and output format.
2. Few-shot examples: 2–10 input/output pairs in the prompt.
3. Retrieval (RAG): supply the facts the model needs at query time (Part 13).
4. Fine-tuning, usually LoRA: teach a format, style, narrow skill or tool-use pattern; distill a big model’s behaviour into a small one.
5. Continued pretraining: a new language or a large specialized corpus.
6. Training from scratch: rarely justified outside labs.

Move down a rung only when an evaluation set shows the rung above falls short.

Prompting from first principles

The model computes p(next token | everything in context). A prompt is not a command channel; it is the conditioning context, and everything in it shifts that distribution - tone, order, examples, even irrelevant text.

• Specify the output: format, length and audience are part of the condition.
• Examples are powerful and literal: the model copies their structure, length and even their mistakes.
• Let the model think before answering: each generated token buys one more forward pass of computation, so reasoning before the final answer gives the model more serial compute. Reasoning models are trained to do this.
• Order matters: stable content first (it helps prefix caching), the specific question near the end.

In-context learning

Large models learn tasks from examples in the prompt with no weight update. Mechanistically, part of this comes from induction-style circuits that copy patterns from earlier in the context (Part 15); some theory frames it as implicit learning inside the forward pass. Its limits: context cost on every request, sensitivity to which examples you choose and their order, and a ceiling on how much a prompt can teach.

Full fine-tuning

Continue training all the weights on your data. It’s the most flexible option, but it needs the ~16 bytes per parameter of training memory (Part 8), produces a full model copy per task, and can cause catastrophic forgetting of general abilities. Use a low learning rate and keep a general evaluation alongside your task evaluation.

LoRA: low-rank adaptation

Freeze the pretrained weight W (d × k) and learn an update ΔW as the product of two thin matrices:

• With d = k = 4096 and r = 16, the update has 2 × 4096 × 16 ≈ 131k parameters instead of 16.8M: under 1%.
• B starts at zero, so training begins exactly at the original model.
• Only A and B need gradients and optimizer state, so memory drops sharply.
• After training, merge BA into W for zero extra inference latency, or keep adapters separate and hot-swap many on one base model.

Why it works: the change needed to adapt a pretrained model appears to have low intrinsic rank. Adaptation mostly steers existing capabilities rather than building new ones. LoRA is usually applied to the attention projections and often the MLP, with r between 8 and 64.

QLoRA

Load the frozen base model in 4-bit (NF4, a format matched to normally distributed weights), keep the LoRA adapters in BF16, and backpropagate through the quantized weights; paged optimizers absorb memory spikes. This made fine-tuning a 65B model possible on a single 48 GB GPU (Dettmers et al., 2023). Related methods: DoRA, adapters, prefix tuning, prompt tuning.

What fine-tuning is good and bad at

Good for	Poor for
Consistent format, structured output, tone	Reliably adding many new facts (use retrieval)
Narrow, repeated tasks at lower cost and latency	Tasks the base model can’t do at all
Tool-calling conventions specific to your system	Information that changes often
Distilling a large model’s behaviour into a small one	Problems you haven’t measured

Fine-tuning on new facts tends to teach the form of an answer more readily than the facts themselves, and can increase confident hallucination about things the model doesn’t know.

Distillation

Train a small student to imitate a large teacher. Classic distillation matches the teacher’s full output distribution (soft labels, softened with a temperature), which carries more information than one-hot labels: the teacher’s second choices encode similarity. Sequence-level distillation fine-tunes the student on texts the teacher generated. Many strong small models are distilled, and this is also how reasoning behaviour transfers from large reasoning models to small ones.

Data and evaluation for adaptation

• Hold out an evaluation set before you start, built from real inputs.
• Clean, consistent examples beat volume; a few hundred to a few thousand often suffice for format and style.
• Watch for leakage between training and evaluation data.
• Compare against a strong prompting baseline; a fine-tune that doesn’t beat it isn’t worth maintaining.

Comparison

Approach	What changes	Data needed	Cost	Best for
Prompting / few-shot	the input only	a few examples	tokens per request	fast iteration, most tasks
RAG	the input, from an index	your documents	indexing and retrieval	knowledge, freshness, citations
LoRA / QLoRA	small added weights	hundreds to thousands of examples	low	format, style, narrow skills
Full fine-tuning	all weights	thousands or more	high	deep behaviour change
Distillation	a new, smaller model	many teacher outputs	medium to high	cutting serving cost

Questions you should now be able to answer

• Why does LoRA initialize B to zero?
• How many trainable parameters does LoRA with r = 8 add to one 4096 × 11008 matrix?
• Why does QLoRA keep the adapters in 16-bit while the base is 4-bit?
• A client wants the model to “know” their 10,000-page wiki. Fine-tune or RAG? Defend it.
• What does a teacher’s soft distribution teach a student that hard labels don’t?