The tools in a data stack for digital pathology, filed by the job each one does, with one question traced through three ways of putting them together
A data stack does about ten jobs: getting data in, moving it, keeping it, crunching it, scheduling the work, packing and running it, shipping changes, reaching across systems, analysing, and showing the results. This guide files common tools under each job, old and new, open source and commercial, says in a line or two what each one is, and gives an example of where it could sit when the data is about pathology slides. The examples are illustrations, not a description of any real lab's setup, and the tools are a sample: every job has more.
The same parts can be put together in different orders. Where the data gets copied decides when an answer shows up and what breaks when a source goes down.
Question No. 1
"How many slides failed scan quality control today, by scanner and by stain?"
The QC result comes from the scanner side (the image store). The stain comes from the lab information system (LIS). So the answer needs both. Three ways to build it, drawn as three lanes below:
Move the sliders and watch where the red arrow lands in each lane.
For two hours. The nightly job starts at 02:00, so try an outage that covers it.
Between 07:00 and 19:00, summed over everyone looking at it.
| Design | Parts, in order | Failure first shown | During the outage | Queries sent to the LIS a day | Copies of the data |
|---|
Made-up timings: the nightly job starts at 02:00 and takes 30 minutes, and a failed run is retried every 30 minutes; the stream shows an event about 2 minutes after it happens and refreshes its copy of the LIS stain list hourly; the virtual layer asks the sources every time the dashboard refreshes, with no cache. Real systems vary on all of these. The point is the shape: where the copy sits decides the delay and the failure mode. Numbers in the table are the part numbers in the catalogue below.
Things to try. Set refreshes to 60 and compare the query column: only the virtual layer's count climbs. Move the outage to 12:30 and the failure to 13:00: the virtual lane's arrow jumps past the outage, while the stream keeps going. Move the outage over 02:00: only the nightly lane notices.
The stream lane is the subject of a companion page, Tape and Tally, which runs a day of scanner events through a partitioned log and a nightly batch side by side.
Not tools to pick so much as the systems everything else reads from.
A scanned slide is a very large image stored as a pyramid: full resolution at the bottom, smaller overviews above, each level cut into tiles so a viewer loads only what is on screen. Scanners often write their own file formats; DICOM, the medical imaging standard, added a whole-slide format in 2010 (Supplement 145).
ExampleEach scan's file location, scanner, and QC result become rows that the rest of the stack can join on.
DICOM WSI. See also Slide to Screen.
The LIS tracks each case by its accession number, and within it each part, block, and slide, with stains and the final report. It is the record of what a slide is.
ExampleThe source of truth for which stain a slide carries and which case it belongs to.
DICOM PS3.17, Annex NN. Practise querying that shape in Rows in the Drawer.
An API is a documented way for one program to ask another for data or to make it do something, often over HTTP with data sent back as JSON. Connecting systems through their APIs means writing the code that calls one system and hands the result to another, handling paging, retries, and errors.
ExampleA small Python job that asks an image store's API for yesterday's scans and writes them to a table.
Getting data out of the sources and into somewhere it can be worked on, either as a steady stream or in loads.
An event streaming platform: programs write events to named topics, Kafka keeps them in order within each partition of a topic, and any number of readers can read them, now or later. It runs as a distributed, fault-tolerant cluster.
ExampleEach scanner sends a "slide scanned" event, keyed by scanner, to a topic that a streaming job and an archive both read.
SQL Server Integration Services: Microsoft's platform for data integration and ETL (extract, transform, load). Packages are built in graphical tools from ready-made tasks and transformations, and stored and run from an SSIS catalog database.
ExampleA nightly package that copies the day's cases from the LIS database into a reporting database.
Connector tools for the extract-and-load half of the job. Each ships a catalogue of ready-made connectors that copy data from apps and databases into a warehouse, leaving the transforming for later (ELT). Airbyte is open source and can be self-hosted; Fivetran is a managed service.
ExampleCopying a scheduling system's tables into the warehouse every hour without writing the API code by hand.
Where the copied data lives so it can be queried with SQL without slowing the source systems down.
A cloud data platform. Data sits in one central store; queries run on "virtual warehouses", separate compute clusters that can be started, stopped, and resized on their own, so two teams' workloads do not slow each other down.
ExampleA table of every scan and QC result, joined to case and stain, for reporting.
Google Cloud's serverless data warehouse. Its storage layer and compute layer scale independently, and users do not provision servers; they load data and run SQL.
ExampleThe same reporting table, if the stack lives on Google Cloud.
Data warehouse software from the Hadoop world (Hadoop being an older system for spreading files and jobs across many machines): it puts table structure over files in distributed storage and lets people query them in a SQL dialect. Its metastore, which records each table's schema and location, is also used by other tools.
ExampleTables over years of tile-level results stored as files in a data lake.
Amazon's fully managed, petabyte-scale data warehouse service. AWS handles provisioning, backups, and patches, and a serverless option charges nothing while the warehouse sits idle.
ExampleThe reporting table again, on AWS.
Engines and languages that transform and summarise data too large for one machine, in batches or as a stream.
A general-purpose engine for processing large data sets across many machines, with APIs in Scala, Java, and Python. The same DataFrame code (tables handled as objects in a program) can run as a batch job or, through Structured Streaming, over a stream, which Spark processes as a series of small batches by default.
ExampleCounting tiles and QC failures per slide across a year of scans, or the live hourly tally in the plate's stream lane.
A dataflow environment for very large files. Scripts in its language, Pig Latin, describe a series of steps (load, filter, group, join), which Pig compiles into MapReduce or Tez jobs on Hadoop. Spark and Hive cover overlapping ground.
ExampleAn older Hadoop job that groups scan logs by scanner and day.
Most of the parts in this guide take one or both. Snowflake, BigQuery, Hive, and Spark all accept SQL; Spark, Airflow, and Dash are driven from Python. Other languages turn up around the edges: JavaScript in front ends, JVM languages in older services and in the tools themselves.
ExampleA Python script that calls an API, cleans the result, and loads it with SQL.
SQL practice on lab-shaped tables: Rows in the Drawer
A framework and distributed engine for stateful processing of both kinds of data: streams with no end, handled as events arrive, and finite data sets, handled whole.
ExampleThe live hourly tally, built stream-first.
A command-line tool for the transform step inside the warehouse. Each model is a SQL SELECT statement; dbt runs the models in order of their dependencies, and adds tests and documentation, with the project kept in Git.
ExampleA model that turns raw scan rows into one clean row per slide, with a test that slide ids are unique.
Something has to start the nightly job, wait for its inputs, retry it when it fails, and say what ran.
A platform for writing workflows as code, then scheduling and monitoring them. A workflow is a DAG (directed acyclic graph) of tasks written in Python; the scheduler runs each task once the tasks it depends on are done, and a web interface shows what ran and what failed. Its own docs say it is not a streaming system.
ExampleEvery night at 02:00: wait for the LIS extract, run the Spark job, check the row counts, refresh the dashboard's table.
Airflow documentation; the streaming note is in the 2.3 docs
Google Cloud's managed Airflow: Google runs the Airflow environment, and you write the same DAGs. Google's documentation now calls it Managed Service for Apache Airflow.
ExampleThe same nightly DAG, without running the Airflow servers yourself.
Two newer Python orchestrators. Dagster is organised around data assets, the tables and files a pipeline produces, rather than around tasks. Prefect turns ordinary Python functions into scheduled, retryable flows with @flow and @task decorators.
ExampleThe same nightly job, written as a chain of assets.
Making a job run the same way on a laptop, a test server, and production, keeping it running, and writing the machines it runs on down as code.
Packs a program into an image that contains everything it needs to run (code, libraries, settings) and runs it as a container: a process kept isolated from everything else on the machine.
ExampleA tiling script and its exact library versions, packed so it runs the same on every machine.
Open-source software that starts, scales, and looks after containerised programs across many machines; a graduated project of the Cloud Native Computing Foundation.
ExampleRunning twenty copies of the tiling container overnight, and restarting any that crash.
An infrastructure-as-code tool. Cloud resources such as servers, storage buckets, and databases are written down in declarative configuration files, and Terraform works out the changes needed to make the real thing match, using plugins called providers for each platform.
ExampleThe pipeline's storage, warehouse, and Kubernetes cluster, defined in files kept in Git.
CI/CD (continuous integration and delivery): every change to pipeline code gets built, tested, and deployed the same way, by a machine.
An open-source automation server with a large plugin ecosystem, used to build projects, run tests, and deploy. You run the Jenkins server yourself.
ExampleOn every merge, run the pipeline's tests against a small sample of made-up slides, then deploy.
GitHub's CI/CD platform. Workflows are YAML files kept in the repository's .github/workflows folder, and run on events such as a push or a pull request.
ExampleThe same tests, run on each pull request, with the result shown on the request.
Not a tool: a way of working modelled on DevOps, aiming to make data work more automated and consistent, with continuous testing and monitoring of pipelines so they keep delivering timely, accurate data. Definitions vary.
ExampleA check that fails the nightly run if any patient appears in both a model's training and test tables.
Querying several systems as if they were one, without copying them first. Several products do this; two of them below.
A data virtualization platform: a logical layer that presents views over many sources and fetches from them when a query arrives, instead of copying the data into a warehouse first. Denodo's own material describes a cache that keeps local copies of source data, to spare busy sources and to cope with slow ones.
ExampleOne view that joins the LIS and the image store, so a report reads both live. The plate's third lane.
Denodo: data virtualization; ISG Research; Denodo blog on caching
A distributed SQL query engine that stores no data of its own. Connectors let one query read data lakes, warehouses, and operational databases where they are, and join across them. A cousin of data virtualization, aimed at large analytic queries.
ExampleOne SQL query joining tile results in a data lake with case data in a relational database.
Statistics and models, once the data is clean and in one place.
An integrated suite from SAS Institute for statistical analysis, data management, business intelligence, and predictive analytics, with its own programming language.
ExampleA statistician's analysis of turnaround times across years of cases.
Models for slides mostly learn from tiles, thousands per slide. The bookkeeping matters as much as the model: tiles from one patient must stay on one side of a train/test split, or the score measures memory.
ExampleA patient-wise split, checked with SQL before training starts.
Bussola et al., AI slipping on tiles; drawn in Same Patient, Both Sides
A free software environment for statistical computing and graphics, with its own language and thousands of add-on packages on CRAN.
ExampleSurvival curves or a mixed-effects model on case outcomes.
Dashboards bought ready-made, and libraries for building your own.
A visual analytics platform: Tableau Desktop for building views, Tableau Prep for preparing data, and Tableau Server (self-hosted) or Tableau Cloud (hosted) for sharing them.
ExampleA QC dashboard by scanner and stain, read from the warehouse.
Microsoft's business analytics platform for turning data into reports and dashboards, now part of Microsoft Fabric.
ExampleThe same dashboard, in a Microsoft shop.
An analytics platform built around asking questions of the data in plain language and getting charts back, rather than building each report by hand.
Example"Failed scans by stain last month" typed into a search box.
Plotly's open-source Python framework for data apps: the layout and the callbacks are written in Python, and Dash is built on Plotly.js, React, and Flask.
ExampleA small internal app that shows a slide's QC history next to its tiles.
A JavaScript library for custom data visualizations. It has no ready-made charts; you build a picture from scales, shapes, and layouts, which suits drawings no chart tool offers.
ExampleA custom view of a slide's tile grid coloured by model score.
A JavaScript library for building user interfaces out of components, used on the web with react-dom and on phones with React Native. Dash uses it underneath.
ExampleThe front end of a slide-review tool, with a D3 chart inside one of its components.
A BI platform, now part of Google Cloud, built around LookML: a modelling language that defines measures and joins once, in version-controlled files, so every report uses the same definitions.
ExampleOne shared definition of "failed scan" that every dashboard reuses.
An open-source data exploration and visualization platform, started at Airbnb and now an Apache project, with a SQL editor for analysts and drag-and-drop charts for everyone else.
ExampleA self-hosted QC dashboard, where licence costs matter.
Overviews of the "modern data stack", such as IBM's and Data Vidhya's lesson, sort tools into layers such as ingestion, storage, transformation, orchestration, and analysis or BI (the exact layers differ between them). This guide uses a similar sorting, tied to slide data, and adds the plate: one question run through three assemblies of the same parts.
Descriptions are this page's own words, based on each maker's documentation or the named source. Names identify the products; this page is not connected with any of them.