12. Inference and serving

Serving is where most AI systems spend most of their money over their lifetime. It is a queueing and memory-management problem, built on the physics from Parts 4 and 5. The first decision is whether to call a model API or run the model yourself.

Kinds of inference

• Online (real-time): a user is waiting, so latency matters most.
• Streaming: tokens appear as they are generated, so perceived latency is the time to the first token.
• Batch (offline): scoring millions of items where nobody waits. Throughput and cost matter; several providers offer batch APIs at about half price.
• Asynchronous: a request is queued and the result is delivered later, by polling or a webhook. The natural fit for long agent tasks.

The metrics that matter

Metric	Meaning	What drives it
TTFT (time to first token)	Wait before the first token appears	Queueing, then prefill of the prompt
TPOT / ITL (time per output token)	Gap between tokens while streaming	Decode speed: memory bandwidth and batch size
End-to-end latency	TTFT + TPOT × output tokens	Both of the above
Throughput	Total tokens per second across all users	Batch size and hardware
Goodput	Requests per second that meet your latency target	Everything together
p50 / p95 / p99	Median and tail percentiles	Users remember the tail, so report it

The central trade-off: larger batches raise total throughput and lower cost per token, but each user’s tokens arrive more slowly. Every serving configuration is a choice on that curve.

What an inference engine actually does

A naive server runs one request at a time and wastes almost the entire GPU, because decoding is memory-bound (Part 4). Modern engines exist to fix that:

• Continuous batching. Requests join and leave the running batch at every decoding step, instead of the whole batch waiting for its longest member. This alone multiplies throughput several times.
• Paged KV cache (PagedAttention). The KV cache is stored in small fixed-size blocks, like an operating system’s memory pages. Fragmentation disappears and many more sequences fit at once. This was vLLM’s founding idea.
• Prefix caching. KV cache for a shared prefix (a long system prompt, few-shot examples, earlier turns of a chat) is reused instead of recomputed. Model APIs sell the same mechanism as discounted “prompt caching.”
• Chunked prefill. Long prompts are processed in slices so they do not freeze token generation for everyone else.
• Quantization. FP8 or 4-bit weights and an FP8 KV cache mean fewer bytes per step, so faster decoding and room for more users.
• Speculative decoding. A small draft model or extra prediction heads propose several tokens; the big model checks them all in one pass. It uses the idle compute of memory-bound decoding and can make generation 2–3 times faster for a single user.
• Parallelism. Tensor parallelism splits a large model across the GPUs of one server; expert parallelism spreads mixture-of-experts models; independent replicas add throughput.
• Disaggregated prefill and decode. Compute-bound prefill and memory-bound decode run on separate GPU pools, with the KV cache transferred between them. NVIDIA Dynamo, llm-d, vLLM and SGLang support variants of this.
• Structured output. Constrained decoding forces output to match a JSON schema or grammar.

The engines you will meet: vLLM (the most widely used open-source engine, with an OpenAI-compatible API), SGLang (fast, strong at prefix caching and structured generation), TensorRT-LLM (NVIDIA’s most optimized path, more setup), NVIDIA Dynamo (a framework for distributed serving across many GPUs), and llama.cpp / Ollama for laptops and CPUs. For embedding models, Hugging Face’s Text Embeddings Inference is a common choice.

Autoscaling GPU inference

• Scale on the right signal. CPU usage means nothing here. Use queue length, waiting requests, KV cache usage, or TTFT against your target.
• Understand the cold start. A new replica waits for node provisioning (1–5 minutes), the image pull (several GB), model loading (weight bytes divided by bandwidth, Part 5), and engine warm-up. Shorten it with pre-pulled images, weights on local NVMe or a fast streaming loader, and warm-up at deploy time.
• Scale to zero only when users can tolerate the wait. Otherwise keep a minimum number of replicas and accept the idle cost, or put a queue in front so bursts wait seconds rather than fail.

API or self-host?

Factor	Model API	Self-hosted open model
Model quality	Frontier proprietary models are only available this way, directly or through Bedrock, Vertex AI or Azure	Open-weight models (Llama, Qwen, Mistral, DeepSeek, Gemma, gpt-oss and others)
Cost shape	Per token; nothing when idle	Per GPU-hour; cheap only at high, steady use
Data control	Vendor processes the data under a DPA; EU processing options vary	Data never leaves your network
Customization	Prompting, some fine-tuning offers	Any fine-tune, LoRA adapters, custom decoding
Operations	Almost none; you handle rate limits and outages	You own capacity, upgrades and on-call

Break-even example, with prices to recheck: you generate 2 billion output tokens per month with an 8B-class model. One H100 at $2.50/hour costs about $1,800 per month and could produce around 6 billion tokens at good batching. Hosted APIs for models of this size have often charged well under $1 per million tokens, so the same 2 billion tokens might cost a few hundred dollars. The API wins, before even counting your time.

Self-hosting wins when volume is very high and steady, when you need a custom fine-tuned model, when data must stay in your network, or when nobody hosts the model you need. Middle grounds exist: dedicated endpoints from GPU clouds and inference providers, and open models hosted inside Bedrock or Vertex AI.

The architecture of an LLM application

Most production AI applications share the same building blocks:

[embedded content: LLM application architecture · 2 paths, 9 components]

The request path runs while a user waits; the ingestion path prepares documents in the background. They meet at the vector index, which is why its freshness and latency matter to both.

• Model gateway. One internal API in front of all models (LiteLLM, Portkey, or a cloud gateway). It centralizes keys, rate limits, fallbacks when a provider fails, response caching, and cost tracking per team or feature.
• Ingestion pipeline for RAG (retrieval-augmented generation), run asynchronously: raw documents land in object storage, a queue triggers workers that parse, chunk and embed them in batches, and results are written to a vector index with metadata.
• Query path: embed the question, retrieve candidates (vector search, often combined with keyword search), rerank, assemble the prompt, generate, and cite sources.
• Agents loop model call, tool call, tool result, model call. That has infrastructure consequences: tasks run for minutes, so they need durable workflow state (Temporal, AWS Step Functions) instead of one HTTP request; code execution needs sandboxes (Firecracker microVMs, gVisor, hosted sandbox services); tools need their own narrow permissions; and every step needs tracing plus step and spending limits. MCP (Model Context Protocol) has become the common standard for connecting tools and data sources to models.
• Rate limits. Providers cap requests and tokens per minute. Handle HTTP 429 responses with exponential backoff, queue work, and spread load across regions or providers through the gateway.
• Caching. Exact-match response caching is safe; provider prompt caching cuts the cost of repeated prefixes; “semantic” caching of similar questions saves money but can return wrong answers, so use it carefully.

Check yourself

• Users say your chatbot takes forever to start answering but streams quickly after that. Which metric is bad, and what are three likely causes?
• Why does continuous batching beat waiting for a full batch on chat traffic?
• Under what conditions does self-hosting an 8B model beat a hosted API on cost?
• Design the ingestion pipeline for one million PDFs: which cloud services would you use, and where would the bottlenecks be?