one day of scanner events, read live and read nightly
Every time a slide scanner finishes a slide, it can send a small message: this slide, this scanner, this time. A batch job collects a whole day of those messages and counts them once, overnight. A streaming job counts them as they come in. This page runs one made-up day of those messages through both, so you can watch what the stream has to handle that the batch job mostly does not: partitions, readers that fall behind or crash, messages that turn up late, and messages that get read twice.
Press Play. The clock runs from 06:00 to 02:30 the next morning. Four scanners at two made-up sites send events into a log split into 3 partitions (the tapes). Consumers read the tapes and keep an hourly tally. At 02:00 a nightly batch job counts the whole day from the same log. Things to try, and what the words mean, are below the wire room.
Scanners
The log: one tape per partition
Consumer group
Hourly tally, by when each slide was scanned
| Scans made so far | |
| Counted by the stream | |
| Dropped as too late | |
| Events read twice | |
| Watermark | |
| Last window published | |
| Events waiting (total lag) | |
| Nightly batch |
Each consumer handles 1 event a minute, a made-up figure. Above 15 per scanner per hour, a single consumer falls behind.
How long the stream waits for stragglers before it publishes an hour and stops accepting events for it.
Off: every message read is counted. On: a message read a second time is recognised by its id and skipped.
Wire log
Simplified: four scanners, three partitions, consumers that each handle a fixed 1 event a minute, offsets committed every 15 minutes, a rebalance that stops everyone for 2 minutes, and one global watermark taken from the newest event time seen. Real systems tune all of these. "Crash one" always picks the consumer with the most uncommitted work, the worst case for replays. The batch job here reads the complete log and removes repeats, so it is always right; real batch jobs have their own bugs. The scan counts are random but repeatable: Reset gives the same day again.
The log here works like Apache Kafka's. A topic is a named log of events, split into partitions so several machines can share it. Each event in a partition gets the next number, its offset. Events with the same key (here, the scanner id) always go to the same partition, and a partition is read in the order it was written; there is no order across partitions.1 Three partitions and four scanners means one tape carries two scanners, so the load is uneven.
Consumers that share a group id split the partitions between them: each partition goes to exactly one consumer in the group, and one consumer may hold several. A consumer beyond the partition count gets nothing. When a consumer joins, leaves, or dies, the group reassigns partitions, and in the classic version of this rebalance nobody reads while it happens.2 Lag is how far a reader is behind the newest offset in its partition; some tools measure it from the last committed offset instead.7
A consumer records how far it has got by committing an offset. Committing after the work is done means a crash between the work and the commit causes the same events to be processed again: at-least-once delivery. The usual answer is to make the write idempotent, for example by keying it on a stable event id so a repeat changes nothing.3
Event time is when the slide was scanned; processing time is when the message got read. Hourly windows here are by event time, so a late message still belongs to its own hour. A watermark tells the engine how late data may be: Spark's Structured Streaming, for one, takes the newest event time seen minus a delay you choose, keeps a window open until the watermark passes its end, and can drop data older than that.4 A longer delay drops less and publishes later. This page uses that idea in simplified form.
Structured Streaming runs a stream as a series of small batch jobs by default, with latency down to about 100 milliseconds, using the same query you would write for a batch.5 The nightly job is the other end of the same idea: one big batch, run by a scheduler. Airflow, a common choice for scheduling such jobs, describes itself in its documentation (here version 2.3.3) as a workflow tool and says plainly that it is not a streaming system.6 Many setups use both: a stream for the dashboard during the day, and a nightly batch that recounts from the log and corrects whatever the stream got wrong, a pattern often called the lambda architecture.8
Both look closely at the log itself. This page tries putting a nightly batch beside the stream, with event-time windows, late data, and replays, on a day of made-up scanner traffic.