Compute for AI: GPUs and accelerators · Cloud

A GPU is a machine for doing the same arithmetic on huge arrays at once. For AI, the number that usually limits you is not how fast it computes but how fast it can move bytes out of its own memory. Understand that one fact and most of AI infrastructure makes sense.

Why GPUs

A neural network is mostly matrix multiplications. Multiplying an (m × k) matrix by a (k × n) matrix takes 2 × m × k × n floating-point operations (FLOPs), and almost all of them are independent. That is perfect work for massive parallelism.

A CPU has a few dozen powerful cores built for branching logic and low latency on one task.
A GPU has thousands of simpler cores grouped into streaming multiprocessors (SMs). They run the same instruction over many data elements at once and are built for throughput.
Tensor cores are special units inside each SM that do a small matrix multiply-and-add in one step. Most of a modern GPU’s speed comes from them, but only with supported number formats and suitably shaped matrices.

The software stack matters as much as the silicon. CUDA is NVIDIA’s programming platform, with libraries for matrix math (cuBLAS), neural network operations (cuDNN) and GPU-to-GPU communication (NCCL). PyTorch sits on top. This 15-year software ecosystem is why NVIDIA dominates. Alternatives exist: AMD’s ROCm, Google’s XLA compiler for TPUs (used through JAX or PyTorch/XLA), and AWS’s Neuron SDK for its Trainium and Inferentia chips.

Number formats (precision)

Every value in a model takes a fixed number of bytes. Fewer bytes means less memory and more speed on hardware that supports the format, with some risk to accuracy.

Format	Bytes per value	Where it is used
FP32	4	Optimizer master weights; older training
TF32	4 stored, reduced in compute	NVIDIA’s default for FP32 matrix math since the A100
BF16	2	The standard for training and inference today; same range as FP32
FP16	2	Older mixed precision; smaller range, needs loss scaling
FP8	1	Training and inference on H100 and newer; about twice BF16 speed
INT8	1	Quantized inference
FP4 / INT4	0.5	Quantized inference; native FP4 on Blackwell GPUs

Memorize the bytes column. It is the basis of every memory estimate in Part 5.

The two numbers that matter: FLOPs and memory bandwidth

An NVIDIA H100 does about 989 trillion BF16 operations per second (TFLOPS) at peak. Its high-bandwidth memory (HBM) delivers about 3.35 TB/s. Divide one by the other and you get about 300: the GPU can do roughly 300 operations in the time it takes to load one byte.

That ratio defines arithmetic intensity, the FLOPs your code does per byte it moves. This is the roofline model:

Below ~300 FLOPs per byte, you are memory-bound. The compute units sit idle waiting for data. Faster math will not help; moving fewer bytes will.
Above it, you are compute-bound. Now the math units are the bottleneck and you are using the chip well.

Three examples to make this concrete:

Training with large batches does big matrix multiplies where each loaded weight is reused across many examples. Intensity is high, so it is compute-bound. Good.
LLM decoding generates one token at a time. Every weight (2 bytes in BF16) is loaded to do about 2 FLOPs per sequence in the batch. With a batch of 1, intensity is about 1, so the GPU runs at well under 1% of its compute. Batching many users together is the only fix, and it is why inference engines exist (Part 12).
Elementwise operations (activations, normalization) touch every byte once and do almost no math. They are always memory-bound, which is why compilers fuse them into neighboring operations.

A trap in spec sheets: NVIDIA often quotes FLOPs “with sparsity,” which doubles the number. The H100’s 1,979 TFLOPS BF16 figure is the sparse one. Halve it for real dense work, then expect to reach 30–60% of that in practice.

GPU memory, from fastest to largest

Registers and on-chip SRAM (about 256 KB per SM on an H100): tiny and extremely fast.
L2 cache (50 MB on an H100): shared by the whole chip.
HBM (80 GB on an H100): the GPU’s main memory; what “GPU memory” usually means.
CPU RAM across PCIe: large but about 50 times slower to reach than HBM.

HBM capacity decides what fits: model weights, activations, optimizer state and the KV cache. “CUDA out of memory” is the most common wall you will hit. FlashAttention, one of the most important AI optimizations, works by keeping attention computations in on-chip SRAM instead of writing a huge intermediate matrix to HBM. It does the same math while moving far fewer bytes.

Interconnects: how GPUs talk to each other

When a model or its training job spans several GPUs, the links between them become the bottleneck. There are two tiers, and every multi-GPU design is about which work goes over which tier.

Scale-up (inside a server or rack): NVLink connects GPUs directly, through NVSwitch chips so every GPU reaches every other at full speed. About 900 GB/s per GPU on H100, 1.8 TB/s on Blackwell. Rack-scale systems such as GB200 NVL72 stretch one NVLink domain across 72 GPUs.
Scale-out (between servers): a separate network, either InfiniBand or Ethernet with RDMA (RoCE). AWS uses its own Elastic Fabric Adapter (EFA). Typically 400 Gb/s per GPU, about 50 GB/s, so roughly 20 times slower than NVLink.

RDMA (remote direct memory access) lets a network card read and write another machine’s memory without involving either CPU. GPUDirect RDMA goes one step further, moving data straight between GPU memory and the network card. Without these, multi-node training spends its time copying.

GPUs sold on PCIe cards (L4, L40S, consumer RTX cards) usually lack NVLink. They are fine for single-GPU inference and small fine-tunes, and poor for multi-GPU training.

The accelerator landscape (approximate, mid-2026)

Chip	Memory per chip	Memory bandwidth	Where you meet it
NVIDIA L4	24 GB	0.3 TB/s	Cheap inference for small models, video
NVIDIA L40S	48 GB	0.9 TB/s	Mid-size inference, fine-tuning
NVIDIA A100	40 or 80 GB	1.6–2.0 TB/s	Previous-generation workhorse, still widely rented
NVIDIA H100	80 GB	3.35 TB/s	The default training and serving GPU today
NVIDIA H200	141 GB	4.8 TB/s	Large-model inference
NVIDIA B200	~180–192 GB	~8 TB/s	Frontier training and inference
NVIDIA B300	~288 GB	~8 TB/s	Frontier, long-context inference
AMD MI300X	192 GB	5.3 TB/s	Large-model inference at lower cost
Google TPU v7 (Ironwood)	~192 GB	~7 TB/s	Google Cloud only, via JAX or PyTorch/XLA
AWS Trainium2	96 GB	~2.9 TB/s	AWS only, via the Neuron SDK

NVIDIA’s next generation (Rubin) was due around late 2026, so check what has shipped. The pattern across generations is consistent: memory capacity and bandwidth grow faster in importance than raw FLOPs for serving large models.

MIG (Multi-Instance GPU) on A100/H100-class chips partitions one GPU in hardware into up to seven isolated slices, each with its own memory.
Time-slicing lets several processes take turns on a GPU, with no memory isolation.
MPS lets kernels from several processes run at the same time.

These suit development, notebooks and small models. Production LLM serving normally uses whole GPUs.

Reading a GPU instance spec

Take AWS p5.48xlarge as an example: 8 × H100 80 GB, 192 vCPUs, 2 TB RAM, 3,200 Gb/s of EFA networking, and eight 3.84 TB local NVMe drives. Read it like this:

GPU count × memory per GPU - what model sizes fit (640 GB total here).
Interconnect - NVLink inside the box? How many Gb/s between boxes? This decides multi-node training speed.
Local NVMe - fast scratch space for datasets and checkpoints.
CPU RAM and vCPUs - enough to run data loading and preprocessing without starving the GPUs.

The equivalents elsewhere: Google Cloud a3-highgpu-8g and Azure ND H100 v5 are both 8 × H100.

Check yourself

A spec sheet shows 1,979 TFLOPS BF16 for the H100. What number should you plan with for real training?
Why is LLM decoding memory-bound at small batch sizes, and what single knob changes that?
You must serve a 70B-parameter model in BF16 on one chip. Which chips in the table can hold its weights, and which leave room for anything else?
Why does an 8 × L40S server train a large model much worse than an 8 × H100 server, even if their FLOPs looked similar?