7. Architecture families and training objectives

The same block can be wired three ways, differing in which tokens may attend to which and what the model is trained to predict. Encoder-only models read bidirectionally and fill in masked tokens; decoder-only models read left to right and predict the next token; encoder–decoder models do both. Decoder-only models dominate general-purpose AI, but encoders remain the right tool for many retrieval and classification jobs.

Family	Attention	Objective	Examples	Best at
Encoder-only	bidirectional: every token sees every token	masked language modeling	BERT, RoBERTa, DeBERTa, ModernBERT	classification, extraction, embeddings, reranking
Decoder-only	causal: each token sees only the past	next-token prediction	GPT series, Llama, Mistral, Qwen, DeepSeek	generation, assistants, in-context learning
Encoder–decoder	bidirectional encoder + causal decoder with cross-attention	denoising (span corruption) or supervised sequence-to-sequence	original Transformer, T5, BART, Whisper	translation, summarization, speech-to-text

Masked language modeling (encoders)

Hide about 15% of input tokens and train the model to predict them from both sides. BERT replaces a chosen token with [MASK] 80% of the time, a random token 10% of the time and leaves it unchanged 10%, so the model can’t rely on seeing [MASK]. Bidirectional context gives strong representations for understanding tasks. The downsides: only the masked 15% of positions give a training signal, and the model doesn’t generate text naturally.

Next-token prediction (decoders)

Predict token t + 1 from tokens 1…t, at every position at once thanks to the causal mask. Every token is a training target, no labels are needed, and generation is native. Crucially, almost any task can be phrased as text continuation - “Translate to German: … →”, “Q: … A:” - so one model and one objective cover everything, with no task-specific heads.

Encoder–decoder

The encoder builds a bidirectional representation of the input. The decoder generates the output, attending to its own past (causal self-attention) and to the encoder’s output (cross-attention). T5 trained this with span corruption: replace random spans with sentinel tokens and generate the missing spans. It’s a natural fit when input and output are clearly different things, such as audio and text, or two languages.

Why decoder-only won for general AI

• Simplicity: one stack, one objective, one inference path.
• Every token supervises: more learning signal per FLOP than masked modeling.
• Predictable scaling: loss falls smoothly with size and data (Part 8).
• In-context learning: large next-token predictors learned to follow examples and instructions in the prompt, replacing task-specific fine-tuning for many uses.
• Efficient generation: under the causal mask, past computations never change, which enables the KV cache.

Encoders are not obsolete. A 100M–400M-parameter encoder fine-tuned for classification or retrieval is often more accurate per euro than prompting a large LLM, and most embedding models and cross-encoder rerankers (Part 13) are encoder-style.

Hybrids worth recognizing

• Prefix LM: bidirectional attention over a prompt prefix, causal attention over the continuation.
• Mixed objectives (e.g., UL2): one model trained on several denoising and next-token tasks.
• Decoder-based embedders: LLMs adapted to output embeddings, often with bidirectional attention switched on during fine-tuning.

Questions you should now be able to answer

• Draw the attention mask of each family for a 4-token input.
• Why does masked modeling give less training signal per sequence than next-token prediction?
• Why is BERT a poor text generator even though it understands text well?
• You need to sort 50 million support tickets into 20 categories cheaply. Which family, and why?
• What does cross-attention give an encoder–decoder that a decoder-only model achieves another way?