Transformers from First PrinciplesPost 6 of 18 · 18 of 18 written
  1. How to use this guide
  2. 01The machinery: how a neural network computes and learns
  3. 02Making deep networks trainable
  4. 03Turning text into numbers
  5. 04Attention from first principles
  6. 05The Transformer block
  7. 06Position: telling the model where each token is
  8. 07Architecture families and training objectives
  9. 08Pretraining at scale
  10. 09Inference and serving
  11. 10Efficiency variants: MoE, long context and beyond attention
  12. 11Post-training: from text predictor to assistant
  13. 12Adapting models: prompting, fine-tuning, LoRA and distillation
  14. 13Embeddings, retrieval and RAG
  15. 14Beyond text: vision, speech and multimodal models
  16. 15Inside the model: interpretability
  17. 16Evaluation, failure modes and security
  18. 17From model to system
  19. 18Fluency kit

6. Position: telling the model where each token is

Transformers · 6 of 183 min readNov 2026

Attention treats its input as an unordered set, so word order must be injected explicitly. The method chosen decides how well a model handles sequences longer than those it trained on. Today the dominant answer is RoPE, which rotates queries and keys so that attention scores depend on relative distance.

Why order is missing

Without position information, shuffling the input tokens only shuffles the outputs (permutation equivariance). The causal mask leaks a little order - a token can tell how many tokens it sees - but that signal is weak and indirect. Models need explicit position information.

Absolute positions

Sinusoidal (the 2017 original). Add to each token embedding a fixed vector of sines and cosines at geometrically spaced frequencies:

PE(pos, 2i)=sin⁡ ⁣(pos100002i/d),PE(pos, 2i+1)=cos⁡ ⁣(pos100002i/d)PE_{(pos,\,2i)} = \sin\!\left(\frac{pos}{10000^{2i/d}}\right), \qquad PE_{(pos,\,2i+1)} = \cos\!\left(\frac{pos}{10000^{2i/d}}\right)

Picture a clock with many hands turning at different speeds: fast hands tell neighbours apart, slow hands tell distant regions apart. Shifting by k positions is a fixed rotation of each sin/cos pair, so relative offsets are easy for the model to compute.

Learned (GPT-2, BERT). A trainable table with one vector per position up to T_max. Simple and effective, but if training stopped at 2,048 positions, position 2,049 has no vector, so the model cannot extrapolate at all.

Relative positions

What usually matters is distance - “the token two back”, not “token 517”. Relative schemes add a learned bias to each attention score based on the offset i − j; T5 groups offsets into buckets and shares biases for far distances. This generalizes better but complicates fast attention kernels.

RoPE: rotary position embedding

RoPE (Su et al., 2021) splits each query and key vector into pairs of dimensions, treats each pair as a 2D point, and rotates it by an angle proportional to the token’s position m:

(q2i′q2i+1′)=(cos⁡mθi−sin⁡mθisin⁡mθicos⁡mθi)(q2iq2i+1),θi=b−2i/dh\begin{pmatrix} q'_{2i} \\ q'_{2i+1} \end{pmatrix} = \begin{pmatrix} \cos m\theta_i & -\sin m\theta_i \\ \sin m\theta_i & \cos m\theta_i \end{pmatrix} \begin{pmatrix} q_{2i} \\ q_{2i+1} \end{pmatrix}, \qquad \theta_i = b^{-2i/d_h}

The key property: rotate q by mθ and k by nθ, and their dot product depends only on (n − m)θ, because rotations compose and the absolute angles cancel. Attention scores become a function of content and relative offset, while the code stays simple: rotate Q and K right after projection, before the dot product. Values are not rotated.

Each pair turns at a different frequency θᵢ. The base b was 10,000 originally; long-context models raise it (500,000 or more) so the slowest rotations don’t wrap around within the context.

ALiBi and NoPE

  • ALiBi adds no position vectors. It subtracts a penalty proportional to distance from each score, −m·(i − j), with a different slope m per head. Recent tokens are favoured by default, and it extrapolates gracefully to longer inputs.
  • NoPE (no explicit position encoding) works surprisingly well in causal decoders, which can infer position from the mask. Some recent models mix NoPE and RoPE layers.

Extending context length

A model trained on 4k-token sequences has never seen rotation angles beyond 4k positions; feed it 32k and attention breaks down. Fixes, roughly in order of sophistication:

  1. Position interpolation: squeeze 32k positions into the trained 0–4k range (divide m by 8), then fine-tune briefly.
  2. NTK-aware scaling: raise the RoPE base instead, stretching low frequencies more than high ones and preserving local resolution.
  3. YaRN and similar: treat each frequency band differently and adjust attention temperature; widely used for extension.
  4. Train on long data: all of the above need some continued training on genuinely long documents.

Advertised context vs usable context

A model that accepts 1M tokens doesn’t necessarily use them well. Retrieval accuracy often degrades with length and drops for information in the middle of long inputs (“lost in the middle”). Needle-in-a-haystack tests overstate ability; multi-hop reasoning across a long context is much harder. For systems work: test your own task at your real context lengths, and prefer retrieving the right 10k tokens over dumping in 500k.

Questions you should now be able to answer

  • Show that a RoPE-rotated q·k depends only on the position difference. (Hint: R(a)ᵀ R(b) = R(b − a).)
  • Why can’t a model with learned absolute positions process sequences longer than its table?
  • Why are values not rotated in RoPE?
  • What goes wrong when a RoPE model trained at 4k sees position 30,000, and how does interpolation fix it?
  • What evidence would convince you that a model actually uses its full context window?