Data Engineering for AI Systems from First Principles

Data engineering is the work of moving data from where it is created to where it is useful, correctly, on time and at acceptable cost, and keeping it that way while everything around it changes. For AI, "useful" means: a model can be trained on it, evaluated against it, or retrieve it at inference time.

How to use this guide

Every data system, from a cron job to a petabyte lakehouse, answers the same six questions. If you can answer them for a system, you understand it.

1. Representation - what is the data, physically and logically? (bytes, types, schema, format)
2. Storage - where does it live, and what does reading it cost? (disk, object store, database)
3. Movement - how does it get from A to B? (batch copies, streams, APIs, change data capture)
4. Transformation - what compute turns raw data into useful data? (SQL, Spark, Python)
5. Trust - how do we know it is correct, fresh, legal and secure? (tests, contracts, lineage, governance)
6. Serving - who consumes it, and with what latency? (dashboards, training jobs, feature lookups, vector search)

The first principles that recur in every section

• Data has physical weight. It is bytes on a device, moved over a wire. Most performance work is reducing bytes read, moved or scanned.
• Everything fails. Machines, networks, upstream sources and humans all fail. Design so a job can be safely re-run, not so it never fails.
• Time is three different things. When an event happened, when you received it, and when you processed it are different clocks. Many AI bugs live in that gap.
• A schema is a contract. Data without an agreed meaning is noise. Most pipeline breakage is a contract changing silently.
• There are no best tools, only trade-offs. Latency, throughput, cost, consistency and simplicity pull against each other. Every tool picks a point on that surface.
• Reproducibility is non-negotiable for AI. You must be able to say exactly which data produced which model, and rebuild it.

How the guide is organised

Part	Sections	What it gives you
Foundations	1–6	Why it matters, how data is represented and stored, databases, modeling, distributed systems
Processing	7–11	Batch, streaming, architectures, pipelines, quality
AI-specific	12–15	Classical ML data, LLM training data, RAG and AI apps, feeding GPUs
Operating	16–17	Governance and privacy, cost and performance
Navigation	18–21	Tool map, learning path, question bank, glossary

How to study it

1. Read a section for the principle, not the tool names. Tools change every two years; principles have not changed in forty.
2. Answer its “Questions to ask” out loud, as if explaining to a colleague. Where you stall, you have found the gap.
3. Build the matching project in section 19. Fluency comes from having broken something and fixed it.
4. Fluency test: pick any tool you hear about and say which of the six questions it answers and which trade-off it makes. If you can, you can learn it in a day.