13. Reliability and observability

Reliability is designing for failures you cannot prevent; observability is being able to explain what your system is doing from the outside. AI systems need both, plus three extra signals: GPU health, token cost, and output quality.

SLIs, SLOs, SLAs and error budgets

• An SLI (service level indicator) is something you measure, such as the share of requests with a first token in under 2 seconds.
• An SLO (objective) is your internal target for it, such as 99.5% of requests succeeding over 30 days.
• An SLA (agreement) is a contractual promise with penalties, usually looser than the SLO.
• The error budget is 100% minus the SLO. While budget remains, ship freely; once it is spent, slow releases and fix reliability.

Availability	Allowed downtime per 30 days	Per year
99%	7.2 hours	3.65 days
99.9%	43 minutes	8.8 hours
99.95%	22 minutes	4.4 hours
99.99%	4.3 minutes	53 minutes

Availability multiplies along a chain. An app at 99.9% that depends on a model API at 99.5% can reach at most about 99.4%. A fallback provider raises it, but only to the extent their failures are independent, which shared causes such as one cloud region often undermine.

Designing for failure

• Timeouts on every network call; nothing should wait forever.
• Retries with exponential backoff and jitter, only for retryable errors (timeouts, HTTP 429 and 503), and with a cap. Jitter adds randomness so thousands of clients do not retry in sync.
• Idempotency: design operations so doing them twice has the same effect as once, using idempotency keys. Essential for anything that charges money, sends messages or writes data.
• Circuit breakers: after repeated failures, stop calling a dependency for a while and fail fast or fall back.
• Graceful degradation: a smaller model, a cached answer or an honest “try again” beats an error page.
• Bulkheads: separate pools and quotas so one customer or feature cannot exhaust everything.
• Redundancy: multiple AZs by default, multiple regions only for critical paths. Backups count only once you have tested a restore.
• Recovery targets: RPO is how much data you can afford to lose; RTO is how long recovery may take. Both drive backup and replication design.
• Limited blast radius: gradual rollouts, feature flags, and splitting customers into independent cells.

Queues and asynchronous work

A queue sits between producers and consumers. It absorbs bursts, retries failed work, and lets you scale workers independently of incoming traffic.

• Services: Amazon SQS, Google Pub/Sub and Azure Service Bus for task queues; Kafka or Redpanda for replayable event streams; Celery or RQ on Redis for simple Python jobs.
• At-least-once delivery is the norm, so consumers must be idempotent. Example: if an embedding worker processes the same document twice, it should overwrite the same vector, not create a duplicate.
• Visibility timeouts and dead-letter queues: a message being processed is hidden for a while; if the worker dies it reappears; messages that fail repeatedly move to a dead-letter queue for inspection.
• Backpressure: a growing queue is the signal to scale consumers or shed load.
• For GPUs: a queue in front of GPU workers keeps them busy at high utilization and turns traffic spikes into short waits instead of failures.

One formula sizes all of this. Little’s Law says the number of requests in the system equals arrival rate times time in system:

At 20 requests per second, each taking 10 seconds to generate, 200 requests are in flight at once. Size your replicas and KV cache (Part 5) for that number.

Observability: metrics, logs and traces

• Metrics are numbers over time: cheap to store and good for dashboards and alerts. Prometheus with Grafana, or the cloud’s own monitoring, or Datadog.
• Logs are individual events with context. Write them as structured JSON, centralize them, and set retention, since storing logs costs money.
• Traces follow one request across services as a tree of timed spans. OpenTelemetry is the open standard for producing them.

Start with the four golden signals: latency, traffic, errors and saturation. Alert on symptoms users feel, such as an SLO burning too fast, not on every internal cause.

GPU observability

nvidia-smi gives a quick look; NVIDIA’s DCGM exporter feeds proper metrics into Prometheus. Watch SM activity, tensor core activity, memory used, memory bandwidth, power, temperature, clock throttling, and hardware error counters (Xid and ECC errors, NVLink errors).

The trap again: “GPU utilization 100%” only means some kernel was running when sampled. A GPU can show 100% while doing a fraction of its possible work. Judge efficiency by SM and tensor core activity, tokens per second and MFU.

LLM observability

For every request, record: the model, input and output token counts, cost, time to first token, total latency, cache hits, errors, tool calls, retrieved documents, and the final answer. For agents, record each step as a span in one trace.

• Tools: Langfuse (open source), LangSmith, Arize Phoenix, Braintrust and W&B Weave. OpenTelemetry is adding standard conventions for generative AI traces.
• Quality in production: run automatic evaluations on a sample of live traffic, collect user feedback, and watch for shifts in what users ask.
• Cost: dashboards per feature and per customer, so a runaway prompt shows up the same day.
• Privacy: traces contain prompts, which contain personal data. Set retention limits and restrict access.

Incidents

Write runbooks for known failure modes, keep a clear on-call owner, and run blameless postmortems that fix causes, not people. Subscribe to your model providers’ status pages, because their outages are now your outages.

Check yourself

• With a 99.9% monthly SLO, how much downtime do you have, and what should change once it is used up?
• Why must consumers of an at-least-once queue be idempotent? Give an example from a RAG pipeline.
• 30 requests per second arrive and each takes 8 seconds. How many are in flight at once, and what does that tell you about KV cache needs?
• A dashboard shows GPU utilization at 100% but throughput is poor. What do you check next?