How to ride
Two kinds of data travel this line every night. The orders line pulls the day's orders from a web API. The partner line picks up a file a partner company drops off. They meet at a join and end at a report that people read in the morning.
Pick a stop below to see the pipeline at that stage. Then pick a bad night and run it. Each stop adds the piece that answers one kind of trouble, which is roughly how many real pipelines grow: something breaks, and the fix becomes part of the design. The numbers are invented and the order of stops is one common path, not the only one.
The line
Service status, every stop
The same six nights at every stage. Read down a column to watch one kind of trouble get handled. Pick any cell to load it above.
Status is how things stand at 8:00 a.m., when people start reading the report.
What these pieces are called
- Scheduler, or orchestrator. A program that runs steps in order, knows which step waits on which, and retries or alerts when one fails. Apache Airflow's task docs describe per-task retries with a wait between tries (five minutes by default). Dagster and Prefect are other open-source examples.
- ETL and ELT. Extract, transform, load cleans data before it reaches the warehouse. Extract, load, transform lands the raw data first and cleans it inside the warehouse, which keeps the raw copy around for reprocessing. AWS has a plain comparison.
- Data tests. Checks that run on the data, not the code. dbt ships four built-in ones: unique, not_null, accepted_values, and relationships.
- Sensors. In Airflow, a sensor is a step whose only job is to wait for something, such as a file showing up, and fail if it does not happen before a timeout.
- Partitions and backfills. Splitting a table by day (or another key) so each slice can be rebuilt on its own. Dagster's docs say a backfill over many partitions launches one run per partition by default.
- Idempotent. A step you can run twice and get the same result as running it once. A day that is rebuilt replaces itself instead of showing up twice.
Already out there
Plenty of tools draw real pipelines. Kedro-Viz shows pipelines written with the Kedro framework, with filtering and a focus mode. Marquez with OpenLineage is an open-source lineage store and graph view, often used next to Airflow. Datadog's Pipeline Simulation runs sample records through a configured observability pipeline to show what each step does to them.
Those show a pipeline as it is. This page tries something smaller and made-up: one pipeline across time, with each new piece tied to the bad night that tends to bring it in.
Where this is rough
- Every number, time, and row count is invented to make the differences easy to see.
- Real teams add these pieces in many orders. Some start with a scheduler on day one; some add data tests before anything else.
- Stage 2 assumes its steps still hand data along in memory, so a rerun starts from the top. Many schedulers can rerun a single failed step if the steps save their output somewhere, which is what stage 6 adds.
- Plenty is left out: schema changes, access control, costs, streaming instead of nightly batches, and the people who get paged.