Transformers from First PrinciplesPost 17 of 18 · 18 of 18 written
  1. How to use this guide
  2. 01The machinery: how a neural network computes and learns
  3. 02Making deep networks trainable
  4. 03Turning text into numbers
  5. 04Attention from first principles
  6. 05The Transformer block
  7. 06Position: telling the model where each token is
  8. 07Architecture families and training objectives
  9. 08Pretraining at scale
  10. 09Inference and serving
  11. 10Efficiency variants: MoE, long context and beyond attention
  12. 11Post-training: from text predictor to assistant
  13. 12Adapting models: prompting, fine-tuning, LoRA and distillation
  14. 13Embeddings, retrieval and RAG
  15. 14Beyond text: vision, speech and multimodal models
  16. 15Inside the model: interpretability
  17. 16Evaluation, failure modes and security
  18. 17From model to system
  19. 18Fluency kit

17. From model to system

Transformers · 17 of 184 min readNov 2026

An LLM call is a stateless function: a context goes in, a next-token distribution comes out, repeated until it stops. Memory, tools, agents and “knowledge” are all built by the surrounding system, which decides what to put into the context and what to do with the output. Designing that system - and doing its cost and latency math - is most of AI-systems engineering.

Context engineering

The context window holds everything the model knows about the current task: system instructions, examples, retrieved documents, tool definitions, tool results, conversation history.

  • Budget it: decide how many tokens each part may use. Long, noisy context lowers quality and raises cost.
  • Order it: stable content first (instructions, tools, reference documents) for prefix caching; the specific request last.
  • Compress it: summarize old turns, keep only relevant tool results, retrieve instead of including everything.
  • Separate trust levels: clearly mark untrusted data (Part 16), knowing that marking alone is not a security boundary.

“Memory” in an assistant is the system storing information and putting the relevant parts back into later contexts. The model itself remembers nothing between calls.

Structured output

Systems need machine-readable output. Options, weakest to strongest:

  1. Ask for JSON in the prompt and parse it; occasionally fails.
  2. Provider JSON modes or schema-following features.
  3. Constrained decoding: at each step, mask out tokens that would violate a grammar or JSON schema, so the output is always syntactically valid. Serving engines and libraries support this.

Valid syntax isn’t correct content: still validate values, ranges and references in code.

Tool use (function calling)

  1. The system gives the model tool descriptions: name, purpose and parameter schema.
  2. The model emits a structured tool call instead of (or before) a final answer - a format learned in post-training.
  3. The system, not the model, executes the call and puts the result into the context.
  4. The model continues: more tool calls, or the answer.

Good tools are like good APIs for a new colleague: clear names, focused purposes, informative descriptions, helpful errors, and outputs sized for a context window. The Model Context Protocol (MCP) standardizes how tools and data sources are exposed to models.

Agents

An agent is a loop: the model chooses an action, the system executes it, the result returns to the context, and this repeats until done. Errors compound: if each step succeeds independently with probability p, n steps succeed with roughly pⁿ - 0.95 per step over 20 steps is about 36%.

  • Prefer fixed workflows (predetermined steps with LLM calls inside) when the task allows; use open-ended agents where flexibility pays.
  • Give the agent ways to check its work: run tests, verify results, re-read outputs.
  • Add checkpoints and human approval for irreversible or costly actions.
  • Keep tools few, sharp and well documented.
  • Evaluate whole trajectories, not just final answers, and log every step.

Cost and latency math

Cost per request:

cost=nin⋅pin+nout⋅pout\text{cost} = n_{\text{in}} \cdot p_{\text{in}} + n_{\text{out}} \cdot p_{\text{out}}

Output tokens are typically priced several times higher than input tokens, because decode is the expensive, memory-bound phase (Part 9). Cached input tokens are usually cheaper.

Latency per request:

latency≈TTFT+nout×TPOT\text{latency} \approx \text{TTFT} + n_{\text{out}} \times \text{TPOT}

Example: 600 ms to the first token, then 500 output tokens at 50 tokens/s, is about 0.6 + 10 = 10.6 seconds. Output length dominates. Levers:

  • Shorter outputs, through explicit length or format instructions.
  • Streaming, so users see progress immediately.
  • Prefix caching for long shared prompts.
  • Routing: a small, fast model for easy requests and a large one for hard ones, or a cascade that escalates on low confidence.
  • Parallel calls for independent subtasks.
  • Batch APIs for non-urgent bulk work, usually discounted.
  • Reasoning budgets: thinking tokens are output tokens, so they cost money and time.

Choosing a model

Ask in order: does it pass your evaluation set; what does it cost per successful task (not per token); does latency meet your needs; is the context window enough in practice; what are the deployment constraints (data residency, privacy, on-premises, open weights vs API); what do the licence terms allow; how stable is the version over time?

Observability and operations

  • Log inputs, outputs, tool calls, token counts, latency and cost per request, respecting privacy.
  • Trace multi-step agent runs end to end.
  • Run the eval suite in CI on every prompt or model change, and pin model versions.
  • Watch production for drift: new input types, rising refusal or error rates, user feedback.

Questions you should now be able to answer

  • Why is an LLM call stateless, and how do assistants appear to remember?
  • What does constrained decoding guarantee, and what doesn’t it?
  • An agent takes 30 steps at 97% per-step reliability. What is its rough success rate, and what would you change?
  • Given prices and speeds, estimate the cost and latency of a request with 8k input tokens (6k of them cached) and 800 output tokens.
  • When would you route between two models instead of using the strongest one everywhere?