6. Storage and data

Cloud storage comes in three shapes: block, file and object. AI systems use all three, plus fast local disks. Object storage is the durable home for datasets, weights and checkpoints; local NVMe is the fast cache beside the GPU; shared file systems serve clusters that must read the same files.

The three shapes, plus local disk

Type	How you use it	Think of it as	AWS / Google Cloud / Azure	Typical AI use
Block	A raw disk attached to one VM; you format it	A drive plugged into one computer	EBS / Persistent Disk / Managed Disks	Boot disks, databases
File	A shared folder tree that many machines mount	A network drive	EFS, FSx for Lustre / Filestore, Managed Lustre / Azure Files	Shared datasets and checkpoints in a cluster
Object	An HTTP API: put and get whole files by key	A huge key–file dictionary	S3 / Cloud Storage / Blob Storage	Datasets, weights, checkpoints, logs
Local NVMe	SSDs inside the physical server	The fastest scratch pad	Instance store / Local SSD / Temp disk	Data and model cache; lost when the VM stops

Object storage from first principles

Object storage is the backbone of cloud data, and its rules explain most data-pipeline design choices.

• Data model. A bucket holds objects; each object is bytes plus metadata, found by a key such as datasets/v2/shard-00042.tar. There are no real folders: the slash is just a character, and “listing a folder” means listing keys with a shared prefix.
• Whole-object writes. You cannot edit bytes in the middle of an object; you replace it. Large files upload in parallel parts (multipart upload). In S3 a single object can be up to 5 TB.
• Durability is not availability. S3 is designed for 99.999999999% (“11 nines”) durability by spreading data across AZs, so losing an object is practically impossible. Availability, whether you can reach it right now, is lower, around 99.9–99.99%.
• Consistency. Since late 2020, S3 gives strong read-after-write consistency: once a write succeeds, every reader sees it. Older blog posts about “eventual consistency” are out of date.
• Performance shape. Each request has high latency (tens of milliseconds), but total throughput is enormous if you run many requests in parallel and read byte ranges. S3 supports thousands of requests per second per key prefix. Few large objects with many parallel readers is fast; millions of tiny files is slow and expensive.
• Storage classes. Hot, infrequent-access and archive tiers trade cheaper storage for costlier, slower retrieval. Lifecycle rules move or delete data automatically by age.
• Pricing shape. Roughly $20–25 per TB per month for hot storage, a small fee per thousand requests, and a significant fee per GB moved out to the internet.

The S3 API is the industry’s common language. Google Cloud Storage, Cloudflare R2, Backblaze B2, Hetzner and self-hosted MinIO all offer S3-compatible APIs, so code written for S3 usually works elsewhere. R2’s selling point is no egress fees.

Block storage and parallel file systems

Block storage (EBS and equivalents) is a network-attached disk that lives in one AZ. You choose its performance in IOPS (operations per second) and throughput (MB/s). Snapshots copy it to object storage for backup. Use it for boot disks and databases.

Local NVMe delivers millions of IOPS and several GB/s per drive, but it disappears when the instance stops. Treat it strictly as a cache you can rebuild.

Parallel file systems (Lustre, and commercial ones such as Weka and VAST) stripe each file across many storage servers. Hundreds of GPUs can then read the same dataset at very high combined throughput. They are common in large training clusters, and managed Lustre can lazy-load files from an S3 bucket. Simpler NFS-style services (EFS, Filestore) are convenient for code and configs but too slow for heavy training input.

Databases: choose by access pattern

Pick a database by how you read and write, not by brand.

What you need	Database type	Examples
Transactions, relations, app state	Relational (SQL)	Postgres via RDS, Cloud SQL, Supabase, Neon
Simple lookups at huge scale with steady latency	Key–value or document	DynamoDB, Firestore, MongoDB
Caching, rate limits, sessions	In-memory	Redis or Valkey (ElastiCache, Memorystore)
Analytics over billions of rows	Columnar data warehouse	BigQuery, Snowflake, Redshift, ClickHouse
Open data files queried in place	Lakehouse	Parquet plus Apache Iceberg or Delta Lake, read by Spark, Trino, DuckDB
Similarity search over embeddings	Vector search	pgvector, Qdrant, Pinecone, Weaviate, Milvus, OpenSearch

Vector search from first principles

An embedding is a list of numbers, typically 768–3,072 of them at 4 bytes each, so 3–12 KB per item. Finding the closest vectors exactly means comparing against every stored vector, which gets slow past a few million.

Approximate nearest neighbor (ANN) indexes trade a little accuracy (recall) for large speed-ups. HNSW builds a navigable graph; IVF groups vectors into clusters and searches only the nearest ones; quantization compresses vectors to save memory.

Memory math: 10 million vectors × 1,024 dimensions × 4 bytes = 41 GB before index overhead. A sound default is pgvector inside the Postgres you already run, until scale or latency gives you a concrete reason to add a dedicated system.

Data formats for AI

• Parquet - columnar, compressed and typed. You read only the columns you need. The standard for tabular and text datasets, including most Hugging Face datasets.
• Arrow - the in-memory columnar format behind pandas, Polars and DuckDB, letting tools share data without copying.
• JSONL - one JSON object per line. Simple and readable; fine for small fine-tuning sets, wasteful at scale.
• Sharded archives (WebDataset tar shards, MosaicML Streaming) - bundle many small samples into files of roughly 100 MB–1 GB, avoiding the small-files problem on object storage.
• safetensors - the standard for model weights. Unlike Python pickle files it cannot execute code when loaded, and it can be memory-mapped for fast loading.
• GGUF - the quantized model format used by llama.cpp and Ollama for local inference.

Feeding the GPUs

Training runs at the speed of its slowest stage, so a GPU waiting on data wastes money. Estimate what the pipeline must deliver:

• Text LLM training on one H100 for an 8B model processes about 8,000 tokens/s, only about 30 KB/s of raw text. Storage is rarely the bottleneck; CPU work such as tokenizing on the fly can be.
• Image or video training is different: 1,000 images/s at 150 KB each is 150 MB/s per node, plus heavy CPU decoding.

Patterns that work:

• Pre-tokenize or preprocess once and store the result; never redo it every epoch.
• Stream large shards from object storage with many parallel readers, and cache them on local NVMe.
• Prefetch the next batches while the current one trains, with enough CPU workers for decoding.
• Shuffle at the shard level plus an in-memory buffer, since random access to single samples in object storage is slow.

A telltale symptom: GPU utilization swinging between 100% and near zero every few seconds usually means the data loader cannot keep up.

Checkpoints

A checkpoint is everything needed to resume training exactly: weights, optimizer state, learning-rate schedule, random number generator states, and the data loader’s position. Full state is about 16 bytes per parameter, so a 70B model checkpoint is around 1.1 TB.

• Frequency trades lost work after a crash against time spent saving.
• Asynchronous checkpointing copies state to CPU memory in seconds, then uploads in the background while training continues.
• Sharded checkpoints let each GPU write its own slice in parallel, which PyTorch Distributed Checkpoint does for you.
• Retention: keep the last few plus milestones; let lifecycle rules delete the rest. Convert final weights to safetensors for serving.

Versioning and lineage

Treat data and models as immutable: write datasets/support-tickets/v3/ instead of overwriting v2. Record which data, code commit and config produced which model, using tools such as DVC or lakeFS for data and MLflow, Weights & Biases or the Hugging Face Hub as a model registry. Lineage is how you debug a regression months later, and it supports the documentation duties some providers have under the EU AI Act.

Check yourself

• Why is listing a “folder” of 10 million small objects slow and costly, and how should the data have been stored?
• S3 promises 11 nines of durability but about 99.9% availability. What does each promise mean in practice?
• A multimodal training job shows GPU utilization oscillating every few seconds. What are the first three things you check?
• Why should you refuse to load a model distributed as a pickle file from an unknown source?