5. Back-of-envelope math for AI workloads

Six short formulas let you estimate memory, time and cost for almost any AI workload in a few minutes, before you rent anything. This is the skill that most visibly separates fluent engineers from everyone else in design discussions and interviews.

1. Memory to hold the weights

Model size	BF16 (2 bytes)	FP8 / INT8 (1 byte)	4-bit (0.5 bytes)
1B	2 GB	1 GB	0.5 GB
8B	16 GB	8 GB	4 GB
70B	140 GB	70 GB	35 GB
405B	810 GB	405 GB	~203 GB

Add 10–20% for the runtime, CUDA context and fragmentation. So an 8B model in BF16 needs a GPU with clearly more than 16 GB, and a 70B model in BF16 does not fit on one 80 GB H100.

2. Memory to train

Full training or full fine-tuning with the Adam optimizer in mixed precision costs about 16 bytes per parameter, before activations:

• BF16 weights: 2 bytes
• BF16 gradients: 2 bytes
• FP32 master copy of weights: 4 bytes
• Adam’s two running averages in FP32: 4 + 4 bytes

So a 7B model needs about 112 GB just for this state, more than one 80 GB GPU. That is why large training splits the state across GPUs (FSDP or ZeRO, Part 11).

Activations come on top: the intermediate values saved during the forward pass for use in the backward pass. They grow with batch size × sequence length × model width × layers. Activation checkpointing saves memory by recomputing them during the backward pass, at roughly 30% more compute.

Parameter-efficient fine-tuning changes the math completely:

• LoRA freezes the base model and trains small adapter matrices, often under 1% of the parameters. An 8B model needs about 16 GB for frozen BF16 weights plus small adapter and optimizer state, plus activations.
• QLoRA also stores the frozen base in 4 bits. An 8B model drops to about 5 GB of weights and fine-tunes comfortably on a single 24 GB GPU.

3. Memory for the KV cache

During generation, a transformer stores the attention keys and values of every previous token so it does not recompute them. This KV cache is often the real limit on how many users one GPU can serve.

Worked examples in BF16:

• Llama 3.1 8B (32 layers, 8 KV heads, head dim 128): 2 × 32 × 8 × 128 × 2 = 128 KB per token. An 8,000-token conversation needs 1 GB; 32 of them at once need 32 GB.
• Llama 3.1 70B (80 layers, 8 KV heads, head dim 128): 320 KB per token. One 32,000-token conversation needs 10 GB.

The lesson: with long contexts, the KV cache, not the weights, decides concurrency. Grouped-query attention (fewer KV heads), FP8 KV caches and paged memory management all exist to shrink or pack it.

4. Compute to train

Here N is the parameter count and D is the number of training tokens. The 6 is 2 FLOPs per parameter per token for the forward pass plus 4 for the backward pass. Then:

MFU (model FLOPs utilization) is the fraction of peak compute you actually achieve. Well-tuned large training reaches about 35–55%; a first attempt is often 15–30%.

Worked example: training an 8B model on 1 trillion tokens.

• FLOPs: 6 × 8×10⁹ × 10¹² = 4.8×10²²
• One H100 at 40% MFU: 989×10¹² × 0.4 ≈ 4×10¹⁴ FLOPs per second
• GPU-seconds: 4.8×10²² / 4×10¹⁴ = 1.2×10⁸, which is about 33,000 GPU-hours
• On 256 GPUs: about 130 hours, or 5.5 days
• At $2.50 per GPU-hour: about $83,000 of compute

Now compare a fine-tune of the same model on 50 million tokens: 6 × 8×10⁹ × 5×10⁷ = 2.4×10¹⁸ FLOPs, about 1.7 GPU-hours. Pretraining is a budget decision; fine-tuning is a lunch-money decision. Data preparation and evaluation usually cost more than the fine-tune itself.

5. Inference speed

Generating one token costs about 2 × N FLOPs, but at small batch sizes the real limit is reading every weight from memory once per step:

• An 8B model in BF16 on one H100: 3.35 TB/s ÷ 16 GB ≈ 210 tokens/s ceiling for one user, about 130–150 in practice.
• A 70B model in BF16 split across 4 H100s: 13.4 TB/s ÷ 140 GB ≈ 95 tokens/s ceiling.

With batching, all sequences in a step share the same weight read. Total throughput therefore grows almost linearly with batch size, until the GPU becomes compute-bound or runs out of KV cache memory. That is the whole economics of LLM serving in one sentence.

Prefill (processing the input prompt) is different: all prompt tokens are processed in parallel, so it is compute-bound. An 8B model reading a 2,000-token prompt needs 2 × 8×10⁹ × 2,000 = 3.2×10¹³ FLOPs, about 0.07 s on an H100 at 50% utilization. Long prompts drive time to first token; long outputs drive total time.

6. Cost per million tokens and data movement

Example: an H100 at $2.50/hour serving an 8B model with batching at 2,500 output tokens/s makes 9 million tokens per hour, about $0.28 per million. If the GPU is busy only 10% of the day, the real cost is ten times that, about $2.80 per million. Utilization dominates every self-hosting cost model.

Data movement is just bytes divided by bandwidth:

• Loading a 140 GB model at 10 Gb/s (1.25 GB/s): about 112 s. From local NVMe at 5 GB/s: about 28 s. This is your autoscaling cold start.
• Saving full training state for an 8B model (about 128 GB) to object storage at 5 GB/s: about 26 s per checkpoint.

The estimation template

Use this order every time:

1. Parameters and number format → weight memory.
2. Training or inference? Add optimizer state and activations, or KV cache × concurrent users × context length.
3. Choose a GPU and count: does it fit with 20–30% headroom?
4. FLOPs → time at a realistic MFU, or bandwidth → tokens per second.
5. GPU-hours × price, divided by expected utilization. Add storage, data transfer and your own time.
6. Sanity-check against a published benchmark or someone else’s reported number.

Check yourself

• Can you fully fine-tune (not LoRA) a 3B model on one 80 GB GPU? What decides the answer?
• How many 32,000-token conversations can an H200 serve for a 70B model in FP8 with a BF16 KV cache?
• You rent an H100 for $3/hour but it is busy only 20% of the time. What do you really pay per busy hour?
• Why does doubling prompt length hurt time to first token more than it hurts tokens per second?