An interactive note, with figures that run in the browser
Why tiles cut from a pathology slide are split by patient, not by tile, before a model is scored
Summary. Models that read pathology slides usually learn from small tiles cut out of each slide, thousands per slide. Tiles from one patient resemble each other in ways that have nothing to do with the disease. If the tiles are shuffled into training and test sets one by one, the same patient lands on both sides, and a model can score well by recognising patients. One study measured scores inflated by up to 41% this way.1 The figures below rebuild the effect on made-up data with a very simple model, so you can see which training tiles each prediction leans on, and what happens to the score when the split is done by patient, or when whole hospitals have a look of their own.
A whole-slide image is too large to feed to most models in one piece, so a pipeline cuts it into a grid of small square tiles, often 256 by 256 pixels, and throws away the tiles that are mostly glass. One large pathology model was trained on 1.3 billion such tiles taken from 171,189 slides,2 which works out to around 7,600 tiles per slide on average.
The label usually lives higher up. A diagnosis belongs to a slide, a case, or a patient, not to each tile. Methods such as multiple instance learning, which treats a slide as a bag of tiles with one label for the whole bag, train on those slide-level labels directly: one study built and tested its system on 44,732 slides from 15,187 patients, using only the reported diagnosis for each slide as its label.3 Either way, a data table of tiles has many rows per slide, several slides per patient, and often many patients per hospital. Those levels are what the rest of this page is about.
The figure below makes up 24 patients, 12 with a tumour and 12 without, and 30 tiles for each. A tile here is two numbers, as if a model had boiled the image down to two measurements. The horizontal one carries a little disease signal. Both carry each patient's own look: staining on the day, how the tissue was cut, the patient's own biology. Tiles from one patient therefore sit in a tight little cloud.
The model is k-nearest neighbours with k = 5: find the five most similar training tiles and copy the label most of them have. It is far simpler than the deep networks in the studies, but it makes the mechanism easy to see. Pick a split, then tap any square (a test tile), or use "Next test tile", to draw lines to the five tiles it leaned on. Under a split by tile, most of those lines go to the test tile's own patient.
| Test accuracy, per tile | |
| Neighbours from the tile's own patient | |
| Test patients | |
| Accuracy per patient (majority of tiles) |
Simplified: the tiles, patients, and numbers are invented, there are two features instead of thousands, and the model is k-nearest neighbours. The size of the gap depends on the sliders, not on any real data set. What carries over is the direction of the effect.
Two things stand out when you play with it. First, slide disease signal to zero: the split by patient falls to about a coin toss, as it should, while the split by tile still scores well above it. That score comes entirely from recognising patients. Second, slide how much patients differ to zero: the two splits agree, because there is no patient look left to recognise. Real tiles sit somewhere in between, so the size of the gap here depends on where the sliders are, and real data sets will differ too.
The leak is not an artefact of the toy. The study that measured up to 41% inflation did so on real slides, saw it even under carefully repeated cross-validation (running the train/test split many ways and averaging), and built a tiling pipeline that splits by patient to avoid it.1
Splitting by patient also changes the arithmetic. The current draw's test set is loading. Tiles from one patient are not independent of each other, so a score worked out over thousands of tiles can look far steadier than a handful of patients can support.
That is also why the dots in Fig. 2 spread out more when the split is by patient. Each draw tests a different handful of people, and the score moves in large steps. A bigger study helps with that by adding patients. Adding tiles from the same patients helps much less.
The same thing happens one level up. Each hospital or lab stains, cuts, and scans its own way. A study of slides in The Cancer Genome Atlas found that a model could tell which site a slide came from, that the site signature survived common colour normalisation and augmentation (random changes to training images), and that it skewed accuracy for predicting survival, mutations, and tumour stage.4 The study's fix was a way of building cross-validation folds so that no site has slides in both training and validation.
The leak needs two things: sites that look different, and sites that send different mixes of cases. In the figure below, 10 of site A's 12 patients have a tumour and 10 of site B's 12 do not, and the slider sets how differently the two sites look. The split is by patient every time, so no patient is on both sides. The score still climbs as the sites drift apart, though the disease signal never changes.
Average test accuracy per tile over 30 draws, each holding out 25% of the patients. Made-up data, same model as Fig. 2.
Most of the fix is bookkeeping, and it starts upstream of the model. Every tile row keeps the patient, case, slide, and site it came from, carried through from the lab system rather than parsed back out of a file name. One patient can have several cases and many slides, so the split is decided on patients (or sites) first, and tiles follow. A short query can then guard the split, run every time the data sets are rebuilt.
Most tools have a ready-made splitter for grouped data. In scikit-learn, GroupKFold, StratifiedGroupKFold, and GroupShuffleSplit take a groups array and never put one group on both sides of a split; the stratified one also tries to keep the classes in proportion.5
-- Patients that appear on both sides of the split. Should be 0. SELECT COUNT(*) AS leaked_patients FROM ( SELECT patient_id FROM train_tiles INTERSECT SELECT patient_id FROM test_tiles ) AS both_sides;
# Five folds that keep each patient's tiles together.
from sklearn.model_selection import StratifiedGroupKFold
cv = StratifiedGroupKFold(n_splits=5)
for train_idx, test_idx in cv.split(X, y, groups=patient_ids):
...
Those are the evidence and the tools. This page tries to add something to push on: a toy where the leak can be watched, turned up, and turned off.
This page explains a methods problem with made-up data. It is not a review of any model or product, and not medical advice.