9. Architectures: warehouse, lake, lakehouse

The three big architectures differ on one question: where structure and guarantees are enforced. The lakehouse became the default by adding database-style transactions on top of cheap, open files in object storage.

The data warehouse

A warehouse is a central analytical database: Teradata historically, today Snowflake, BigQuery, Redshift or Azure Synapse. Data is loaded into managed, usually proprietary storage with a strict schema.

• Strengths: fast SQL, ACID transactions, strong governance, little to operate.
• Weaknesses: storage and compute are priced by the vendor, data sits in the vendor’s format, and unstructured data such as images and PDFs fits poorly. ML teams often export data out of the warehouse to train, creating copies.

The data lake

A lake stores everything in object storage in open formats: Parquet, JSON, CSV, images, audio. Any engine can read it.

• Strengths: very cheap, holds any data type, no vendor lock-in, natural home for ML and AI data.
• Weaknesses: plain files have no transactions, so readers can see half-written data. Nothing enforces the schema. Updating or deleting individual rows, which GDPR requires, means rewriting files by hand. Without discipline a lake decays into a “data swamp” nobody trusts.

The lakehouse

A lakehouse keeps the lake’s open files in object storage and adds a table format layer that provides warehouse guarantees:

• ACID transactions: readers never see partial writes.
• Schema enforcement and safe schema evolution.
• Row-level updates, deletes and MERGE (upserts).
• Time travel: query the table as it was at an earlier version.
• Efficient metadata, so engines skip files without listing directories.

Because the format is open, Spark, Trino, Flink, DuckDB, Snowflake and others can read and write the same tables.

How a table format works, from first principles

This is worth understanding deeply, because it combines ideas from sections 3, 4 and 6.

1. Data files are ordinary immutable Parquet files in object storage.
2. Metadata files list exactly which data files make up each version of the table, called a snapshot, plus statistics about each file.
3. A commit writes new data files first, then writes new metadata, then atomically swaps a single pointer to the new metadata. The catalog performs this swap with a compare-and-swap, so two writers cannot both win.
4. Readers pick a snapshot and read only the files it lists. They never see files from an uncommitted write. This is MVCC snapshot isolation, rebuilt on object storage.

Everything else follows. Time travel is reading an older snapshot. Deleting a row means writing a new snapshot whose files exclude it, either by rewriting affected files (copy-on-write) or by recording deletes in small side files and merging them at read time (merge-on-read). Compaction rewrites many small files into fewer large ones. Expiring snapshots finally removes unreferenced old files, which is the step that actually erases data for GDPR.

The formats and catalogs

Format	Origin	Distinctive traits
Apache Iceberg	Netflix, now engine-neutral	Hidden partitioning, partition evolution, REST catalog standard, broad engine support
Delta Lake	Databricks	Transaction log as a folder of JSON commits, deep Spark integration
Apache Hudi	Uber	Optimized for frequent upserts and streaming ingestion
Apache Paimon	Flink community	Streaming-first lakehouse tables
Lance	AI-focused	Fast random access and vectors for multimodal ML data

A catalog maps table names to their current metadata, and often manages permissions. Examples include Hive Metastore, AWS Glue, Databricks Unity Catalog, Apache Polaris, and Project Nessie, which adds Git-like branches to data.

ETL versus ELT

• ETL (extract, transform, load) transforms data before loading it. This made sense when warehouse compute was scarce and expensive.
• ELT loads raw data first, then transforms it inside the warehouse or lakehouse, usually in SQL with a tool like dbt. Storage is cheap, compute is elastic, and the raw data stays available.

For AI work, always keep the raw data. You will re-parse documents with a better parser, re-chunk them, and re-embed them with a newer model. None of that is possible if you only kept the processed output.

The medallion layers

Layer	Contains	Rules	AI examples
Bronze (raw)	Data exactly as received	Append-only, never edited, full history	Crawled HTML, PDFs, API dumps, raw app logs
Silver (clean)	Typed, validated, deduplicated, joined	Rebuildable from bronze at any time	Extracted text with metadata, deduplicated documents, cleaned events
Gold (ready)	Shaped for one use	Rebuildable from silver, owned by a consumer	Training mixtures, eval sets, chunk and embedding tables, feature tables

The value is that each layer can be rebuilt from the one below it. A bug in silver is fixed by fixing code and rebuilding, not by begging a source system to resend data.

Organizational patterns

• Data mesh: domain teams own their data as products with clear interfaces; a platform team provides self-serve infrastructure; governance is federated. It is an organizational design, not a tool you install.
• Reverse ETL: pushing modeled data back into operational tools such as a CRM, so predictions and segments reach the people who act on them.

Questions to ask

• Where is the source of truth, and in what format?
• Can two writers corrupt this table, or does a commit protocol prevent it?
• Can I query the table as it was on the day the model was trained?
• When a user asks for deletion, which files physically change, and when?
• Can every layer be rebuilt from the layer below it?