3. Storage: where bytes live and what reading them costs

Each step down the memory hierarchy is roughly 10 to 1,000 times slower and cheaper than the one above. Almost every storage design rule follows from that: read as little as possible from slow tiers, and read it in large sequential chunks.

The memory hierarchy

The numbers below are approximate orders of magnitude from memory, not measurements of any specific machine. Memorize the ratios, not the digits.

Operation	Approximate latency	If 1 ns were 1 second
CPU L1 cache read	~1 ns	1 second
Main memory (RAM) read	~100 ns	~2 minutes
NVMe SSD random read	~10–100 µs	~3 hours to 1 day
Round trip inside a data centre	~500 µs	~6 days
Spinning disk (HDD) seek	~5–10 ms	~2–4 months
Object storage (S3) first byte	~10–100 ms	~4 months to 3 years
Round trip Europe to US	~100 ms	~3 years

The lesson: a single round trip to object storage costs as much time as millions of memory reads. Systems that touch object storage must batch and parallelize, or they crawl.

Latency versus throughput

• Latency is the time until the first byte arrives.
• Throughput (bandwidth) is how many bytes per second flow once data is moving.
• Sequential access pays latency once and then streams at full bandwidth. Random access pays latency on every read.

Rough bandwidths, again approximate: an NVMe SSD reads several GB/s, an HDD 100–250 MB/s, and a 10 Gbit/s network about 1.25 GB/s. A single connection to object storage often gets tens of MB/s, so tools open dozens of connections in parallel.

Back-of-envelope habit: 10 TB read at 1 GB/s takes about 10,000 seconds, roughly 3 hours. Do this arithmetic before choosing a design; it rules out bad ideas in seconds.

Block, file and object storage

Kind	Model	Strengths	Weaknesses	Examples
Block	A raw disk volume attached to one machine	Lowest latency, supports databases	One machine at a time, you manage the file system	AWS EBS, local NVMe
File	A shared hierarchy of folders and files (POSIX)	Familiar, many machines can mount it	Hard to scale, metadata bottlenecks	NFS, AWS EFS, Lustre
Object	A flat map from key to immutable blob, over HTTP	Practically unlimited, cheap, very durable, decoupled from compute	High latency, no in-place edits, no atomic rename	Amazon S3, Google Cloud Storage, Azure Blob, MinIO

Object storage is the foundation of modern data and AI platforms. It is cheap per gigabyte, designed for extreme durability, and any number of compute clusters can read it at once. That last point is the idea of separating storage from compute: you pay for compute only while it runs.

What object storage changes

• Objects are immutable. You cannot append to or edit a file; you rewrite it. This is why Parquet files are written once and never modified.
• Folders are an illusion. “datasets/2026/10/” is just a shared key prefix. Renaming a folder means copying every object, so there is no atomic rename.
• Listing is slow and requests cost money. Listing millions of keys takes minutes, and every GET and PUT is billed.
• No transactions across objects. Writing 1,000 files is not atomic: a reader can see 600 of them. Table formats (section 9) exist to fix exactly this.

Distributed and parallel file systems

HDFS, from the Hadoop era, splits files into large blocks, stores each block on three machines, and keeps all metadata on a central NameNode. It brought computation to the data. Object storage has largely replaced it in the cloud.

Large GPU training clusters often use parallel file systems such as Lustre, IBM Storage Scale (GPFS) or WEKA. They stripe files across many servers so thousands of GPUs can read at once. Another common pattern stages data from object storage onto each node’s local NVMe before training.

Compression: trading CPU for I/O

Compression pays whenever reading bytes is slower than decompressing them, which is almost always for disk and network.

Codec	Speed	Ratio	Typical use
Snappy / LZ4	Very fast	Modest	Hot data, Parquet defaults in some tools
Zstandard (zstd)	Fast, tunable	Good to very good	A strong general default today
gzip	Slow	Good	Legacy and interchange; a single .gz file cannot be split for parallel reads

Text and logs compress very well. JPEG images, MP3 audio and most video are already compressed, so compressing them again wastes CPU.

Durability, availability and backups

• Durability means the data is not lost. Availability means you can reach it right now. A system can be durable but temporarily unavailable.
• Replication keeps several full copies. Erasure coding stores data plus parity blocks, surviving failures with far less overhead than 3 full copies.
• Replication is not backup. If someone deletes a table, replication faithfully deletes every copy. Backups, snapshots and object versioning protect you from your own mistakes.

Storage tiers and lifecycle

Cloud providers offer hot, infrequent-access, cold and archive tiers. Archive tiers are very cheap to store but can take hours to retrieve and charge for retrieval. Lifecycle policies move old raw data down tiers automatically, which matters once datasets reach hundreds of terabytes.

AI-specific storage arithmetic

• Embeddings: 10 million vectors × 1,024 dimensions × 4 bytes (float32) is about 41 GB, before any index overhead. Using float16 halves it.
• Checkpoints: model weights plus optimizer state are commonly several times the weight size. Writing them must be fast, or GPUs sit idle.
• Training reads: a large run may read its dataset several times. Whether that comes from object storage, a parallel file system or local NVMe decides whether GPUs wait.

Questions to ask

• Is the access pattern sequential or random, and does the storage suit it?
• What does one read cost in latency, and how many reads will the job do?
• How long does it take to read the whole dataset at the available bandwidth?
• What happens if two writers touch the same data at once?
• If someone deletes this by mistake, how do we get it back, and how fast?