Color is data too
Same tissue, three labs, three shades of pink.
Stained tissue gets its color from two dyes: hematoxylin turns cell nuclei purple, eosin turns most everything else pink. No two labs apply them alike. Dye lots, staining times and scanner lamps all differ. A person adapts without noticing. A model trained at one site doesn't, and a stain that drifts at one scanner can quietly break it. This page pulls the two dyes apart, puts them back in a standard color, then watches three days of scans stream in for drift.
Everything here is synthetic. The tissue is procedural and the stains are applied with the same light-absorption physics a scanner sees, so the true dye amounts are known exactly. The estimation, normalization and monitoring code only ever sees the pixels. Nothing leaves your browser.
Pull the colors apart, then put them back
One spot of tissue, scanned at four sites. The top row is what each scanner saw. The bottom row is after Macenko normalization: estimate each patch's two dye colors from its own pixels, work out how much of each dye every pixel holds, and repaint with a reference pair. ΔE is the average color difference from Scanner A; about 2.3 is the smallest difference a person can see.
Make your own scanner
Every pixel is a mix of two dye colors
Each dot is a pixel in optical density, the log of how much light it absorbs, projected onto the plane the two reference dyes span. Pixels fan out between two edges: pure hematoxylin and pure eosin. The lines are each scanner's estimated dyes. A different lab is a different fan.
One dye at a time
The same patch split into its two dyes using that scanner's estimated colors (color deconvolution). Hematoxylin alone is the nuclei; eosin alone is everything else.
Watch three days of scans stream in
Three scanners, five slides an hour each. Every scan becomes an event. The pipeline checks focus, estimates the dyes, normalizes, and hands a patch to a model trained at Site A that flags atypical nuclei by how darkly they stain. A monitor groups events into 4-hour windows per scanner, learns what normal looks like on day one, then runs a CUSUM (cumulative sum) test on every metric to catch both sudden jumps and slow creep. It has no idea what's coming. The dashed lines mark what actually happened, as each moment arrives.
Latest events
The last event, as the topic carries it
Nothing yet. Press Play.
Alerts
How this maps to a real build
What each piece on the page would be in production, on Google Cloud with open-source equivalents alongside.
-
The topic
Each finished scan publishes a small event to Pub/Sub (or Kafka): slide id, scanner, timestamp, storage path. Pixels stay in the bucket. Failed QC goes to a separate rescan topic, so it's a work queue for the lab rather than a silent drop.
-
Streaming job
An Apache Beam pipeline on Dataflow (or Spark Structured Streaming): read the event, read a few tiles, compute focus and stain estimates, key by scanner, window into fixed 4-hour windows, aggregate, and keep the CUSUM in per-key state. Event time, not arrival time, with a watermark for scanners that upload late.
-
Normalization as a versioned transform
The reference dyes and the running per-scanner estimate are inputs with a version, not constants in code. Training data and live data go through the same transform, or the model learns one color and is served another.
-
Same code, batch and stream
Beam runs the identical pipeline in batch over BigQuery history, so a new metric or a fixed bug can be backfilled across a year of scans, and the baselines rebuilt, without a second implementation.
-
Metrics and alerting
Window aggregates land in BigQuery for dashboards (Looker, Tableau, Power BI). Alerts go to Cloud Monitoring and on to the lab's on-call, with the scanner, metric, z-score and a link to example patches.
-
What production adds
Accuracy needs labels, so a small audited sample per scanner per day gets read by a pathologist. Baselines per scanner and per stain protocol. Known events, like a scheduled dye lot change, annotated so they don't page anyone at 3 a.m.
What's simulated
| Piece | On this page | In a real system |
|---|---|---|
| Tissue and dyes | Procedural tissue; dye amounts turned into color with the Beer-Lambert law | Real slides, with real chemistry: dyes aren't perfectly linear, and fading, thickness and folds all change color |
| Stain estimation | Macenko on one 80 × 80 patch, with white measured on bare glass | The same method on many tiles per slide, or a learned method; white from the slide's background |
| The model | One threshold on how dark the nuclei are, learned on Site A | A deep network, which fails the same way for the same reason, just less visibly |
| Accuracy | Against every patch's true label, known because the tissue is generated | Against an audited, labelled sample, so it lags and is noisier |
| The stream | A loop in your browser, 1,080 events over 72 simulated hours | A managed streaming job, with late data, retries and exactly-once sinks |
The engine is separate from the page and has its own tests. Read it in stain-drift/, and the tissue generator in shared/.