16. Fluency toolkit: the questions to ask

Fluency is knowing which question to ask next. These checklists cover the situations you will meet most often; the last section says where trustworthy answers live. For a single unfamiliar service, use the seven questions at the top of this guide.

Designing a new AI system

1. Who are the users, and what latency do they need: interactive, minutes, or overnight?
2. What is the load: requests per second now and at peak, and tokens per request?
3. What model quality is truly needed? Could a smaller model do it? API or self-hosted?
4. Where does the data live, and what legal constraints apply (personal data, EU residency)?
5. What happens when the model provider is slow or down?
6. How will we know it works: which SLOs and which evaluation metrics?
7. What does one unit cost (a request, a user, a document), and what happens at ten times the scale?
8. What is the simplest version that could work, and what evidence would force us to change it?

Planning a training or fine-tuning run

1. What metric and evaluation set define success, decided before any GPU money is spent?
2. Parameters, precision and method (full, LoRA or QLoRA): how much memory, so which GPU and how many?
3. FLOPs: how many hours at a realistic MFU, and what cost?
4. Data: how big, in what format, where, and is preprocessing finished in advance?
5. Checkpoints: how often, where, and has a restore actually been tested?
6. Failures: does the job restart automatically? Spot or on-demand?
7. Tracking: are config, code commit, data version and metrics all logged?

Debugging slow training

1. Is the GPU really busy? Check SM and tensor core activity, not just “utilization.”
2. Is data loading the bottleneck? Run with synthetic data; if it gets much faster, yes.
3. Is communication exposed? Compare single-node and multi-node speed and look at NCCL time in the profiler.
4. Is the batch too small, the precision too high, or FlashAttention and compilation missing?
5. Is one node a straggler? Compare step times per GPU.
6. Is NCCL using the fast network path, not plain TCP?

Debugging slow or expensive inference

1. Is the problem time to first token or time per token? They have different causes.
2. Are requests queueing? Check waiting requests and apply Little’s Law.
3. Is the KV cache full, causing requests to be paused or recomputed?
4. Are batch size, concurrency limits and prefix caching configured well?
5. Is the model larger than the task needs? Would quantization or a smaller model hold quality?
6. For APIs: prompt length, cache hit rate, output length, hidden retries, and agent loops.

Reviewing security

1. Which identities exist, what can each do, and are there any long-lived keys?
2. What is reachable from the internet, and should it be?
3. Where are the secrets, and who can read them?
4. Is data encrypted, and who controls the keys?
5. What untrusted content reaches a model that has tool access?
6. What gets logged, for how long, and who can see it?

Reviewing a cloud bill

1. Are the top five line items what you expected?
2. Is anything running that nobody owns, such as untagged resources?
3. How well used are the largest compute items?
4. What are data transfer and NAT costs?
5. How fast is storage growing?
6. Is cost per unit falling or rising as usage grows?

Where to find trustworthy answers

• Provider documentation and pricing pages are the source of truth for limits, quotas and prices. Pricing calculators help with estimates.
• Architecture guides: the AWS Well-Architected Framework (including its machine learning and generative AI lenses), the Google Cloud Architecture Center and the Azure Architecture Center.
• Technical reports and engineering blogs: the infrastructure sections of model papers (the Llama 3 and DeepSeek-V3 reports are excellent), plus the vLLM, SGLang, PyTorch and NVIDIA technical blogs.
• The tools themselves: source code, release notes and GitHub issues often answer what documentation does not.
• Benchmarks such as MLPerf, read together with their exact configuration.
• Your own small experiment: a 20-minute test on a rented GPU often settles a question faster than a day of reading.

When asking a colleague, a forum or an AI assistant, include your goal, constraints, what you tried, the concrete numbers (model size, GPU type, batch size, exact error message), and what you expected versus what you saw. With AI assistants specifically: ask them to show the estimate step by step, ask what would make the answer wrong, and verify anything version- or price-specific against current documentation.