A six-gigapixel image that opens instantly, because you never read most of it.

From one slide to a training set

A model can't take six gigapixels in one bite, so the slide gets cut into patches. First, find the tissue: three quarters of the glass is empty, and every empty patch you keep is wasted compute and noise in training. The usual first pass is a threshold on a low-resolution thumbnail, picked automatically with Otsu's method. Then lay a grid over it and keep the patches that are mostly tissue.

The output is a table, not a folder of images

Writing every patch out as a PNG copies the slide again, badly compressed. The better output is a manifest: where each patch is, in the pyramid's own coordinates, plus whatever you need to filter on. The training job reads the pixels straight from the pyramid when it needs them.

CREATE TABLE pathology.patches (
  slide_id     STRING NOT NULL,
  patch_id     STRING NOT NULL,
  level        INT64,     -- pyramid level to read
  x            INT64,     -- level-0 pixel coordinates
  y            INT64,
  size_px      INT64,
  tissue_frac  FLOAT64,
  mask_version STRING,    -- which detector made it
  created_at   TIMESTAMP
)
PARTITION BY DATE(created_at)
CLUSTER BY slide_id;

-- A training split is a query, not a copy:
SELECT p.* FROM pathology.patches p
JOIN pathology.slides s USING (slide_id)
WHERE p.tissue_frac >= 0.5
  AND s.qc_status = 'pass';

Now do that for a whole lab

One slide is a curiosity. A lab scans every slide it makes, every day, and keeps them for years. Move the sliders to see what that does to storage, ingest bandwidth and patch counts.

How this maps to a real build

Here's what each piece on the page would be in a production pipeline on Google Cloud.

• The file

  Scanners write tiled, pyramidal TIFF variants (SVS, NDPI, MRXS and friends) or DICOM whole-slide images. OpenSlide or tiffslide read tiles by level and position without opening the rest. Converting to DICOM puts every vendor's format behind one standard, and a DICOM store can serve tiles over DICOMweb.

• Landing and orchestration

  Scanners drop files into a bucket; each new object fires a Pub/Sub message. An Airflow (Cloud Composer) DAG, or an event-driven Cloud Run job, checks the file opens, registers the slide, and fans out the per-slide work.

• Tissue detection and patching

  An Apache Beam pipeline on Dataflow, one element per slide: read the thumbnail level, build the mask, filter ink and artifacts, emit patch coordinates. It's embarrassingly parallel, so a day's slides finish in minutes with enough workers.

• Manifests, not copies

  Patch coordinates go to a BigQuery table, partitioned by date and clustered by slide. Training splits, sampling and class balancing become SQL. The data loader reads pixels from the pyramid on the fly, with a local tile cache, instead of from millions of small files.

• Viewing

  A Deep Zoom or DICOMweb tile endpoint behind a CDN, and a viewer like OpenSeadragon doing exactly what this page does: pick a level, request visible tiles centre-first, draw a coarser level while they arrive, cancel what scrolled away.

• What production adds

  Lifecycle rules that move old slides to colder storage classes. A versioned mask detector, so every patch can be traced to the code that chose it. QC flags (blur, folds, ink) as columns, not deletions. Checksums on landing.

What's simulated

The engine is separate from the page and has its own tests. Read it in slide-pyramid/, and the tissue generator in shared/.