10. Pipelines and orchestration

A pipeline is a graph of idempotent steps over partitioned data. Orchestration is the system that runs that graph in dependency order, on a schedule or on events, retries what fails, and lets you safely re-run history.

A pipeline step as a function

Think of every step as a pure function: output = f(inputs, code version, parameters). The parameter is usually a partition, such as “the data for 2026-10-29”. If a step really behaves this way, it is reproducible, testable and safe to retry. Most pipeline pain comes from steps that secretly depend on something else: the current time, a mutable table, a random seed, or an API that returns different answers.

DAGs

Pipelines are modeled as directed acyclic graphs: nodes are steps or datasets, edges are dependencies, and no cycles are allowed, so a valid execution order always exists.

There are two ways to think about the nodes:

• Task-centric (classic Airflow): nodes are actions. “Run this script, then that one.”
• Asset-centric (Dagster assets, dbt models): nodes are the datasets that should exist. “This table is built from those two tables.”

Asset-centric thinking maps directly to lineage and freshness questions: is this table up to date, and what feeds it? That makes it a better default for data and ML work.

Scheduling and data intervals

• Time-based: run every day at 02:00 (cron).
• Event- or data-driven: run when an upstream partition lands, a file appears, or a dataset updates.

Most orchestrators distinguish the run time from the data interval it covers. The run that starts on 30 October at 02:00 processes the data for 29 October. Confusing the two produces off-by-one-day bugs.

Full refresh versus incremental

Late-arriving data breaks naive incremental logic: an event for 28 October arrives on 30 October. The common fix is a lookback window: each run also reprocesses the previous few partitions.

Backfills

A backfill re-runs a pipeline over historical partitions, after fixing a bug or adding a new column or feature. Backfills are only safe if steps are idempotent and parameterized by partition. Plan their cost and parallelism: backfilling two years of daily partitions is 730 runs, and it can overwhelm a source database or a budget.

Ingestion patterns

• APIs: handle pagination, rate limits, retries with exponential backoff, and incremental cursors.
• Files: other parties drop files into object storage or SFTP. Validate them before trusting them.
• Databases: periodic snapshots for small tables, CDC for large or fast-changing ones.
• Event streams: consume from Kafka or similar into bronze tables.

Tools include Fivetran and Airbyte (many prebuilt connectors), dlt (a Python library for writing your own) and Meltano. For a single unusual source, a small custom Python job is often simplest.

Transformation with dbt

dbt turns SQL SELECT statements into managed tables. Each model is one SELECT; ref() declares dependencies, from which dbt builds the DAG. It adds tests, documentation and lineage, and runs the same code against dev and prod environments.

An incremental model that processes only new events:

-- models/silver/llm_calls.sql
{{ config(materialized='incremental', unique_key='request_id') }}

SELECT
  request_id,
  session_id,
  CAST(ts AS TIMESTAMP)   AS called_at_utc,
  model_version,
  prompt_version,
  input_tokens,
  output_tokens,
  latency_ms
FROM {{ ref('bronze_llm_calls') }}
{% if is_incremental() %}
WHERE ts > (SELECT max(called_at_utc) - INTERVAL 3 DAY FROM {{ this }})  -- lookback for late data
{% endif %}

The unique_key makes the step idempotent: reprocessed rows overwrite themselves rather than duplicating.

Orchestrators

The tool matters less than the discipline. Any orchestrator works if steps are idempotent, partitioned and observable.

Write-audit-publish

A pattern every serious pipeline should use:

1. Write new output to a staging location: a branch, a temporary table or a staging partition.
2. Audit it: row counts, schema, null rates, uniqueness, distribution checks.
3. Publish it atomically only if the audit passes, swapping it into the place consumers read.

Consumers never see bad data, and a failed audit leaves yesterday’s good data in place. Iceberg branches and Nessie make this a first-class operation.

Pipelines are software

• Keep code, SQL and configuration in version control, and review changes.
• Run tests in CI against small sample data.
• Separate dev, staging and prod environments.
• Describe infrastructure as code (Terraform, Pulumi), and package jobs in containers.
• Keep secrets in a secrets manager, never in code or notebooks.

ML pipelines are the same thing

A training pipeline (prepare data, train, evaluate, register the model, deploy) is a DAG with the same rules. The difference is its artifacts: datasets, models and metrics, which must be versioned and linked to each other. Experiment trackers and model registries such as MLflow record those links.

Questions to ask

• Is each step a pure function of its inputs, code and partition?
• What exactly happens if this run is triggered twice?
• How do we handle data that arrives late?
• How long, and how much money, would a full backfill take?
• Can bad output ever reach consumers, or is there an audit gate?