Module 1 · Data & Labels

Overview

A supervised model can only be as good as the examples and labels it learned from. This lesson looks at what goes into a model like PTX-Detect: the kinds of data, who decides the "right answer", and how developers keep some data aside to test the model honestly. About 15-20 minutes.

By the end of this lesson you can:

Name the four kinds of data radiology AI uses.
Explain what a label is, who makes it, and why labels can be wrong.
Explain why data is split into training, validation and test sets.
Recognize data leakage, such as the same patient in training and test.

What radiology AI learns from

Different kinds of data feed different kinds of models. Click through the four main types.

1

Medical images

The main input for most radiology AI, including PTX-Detect

X-rays, CT, MRI, ultrasound and mammograms. To a computer, an image is a grid of numbers, one per pixel, giving its brightness. Hospital images are stored in a format called DICOM, which also carries extra information such as the scanner type and patient position (Lesson 2.3 covers DICOM in detail). Image models are usually convolutional neural networks (Lesson 1.2).

Image needed: chest X-rayAny normal frontal (PA) chest X-ray.

Image needed: CT sliceOne axial chest or head CT image.

Examples

Chest X-ray tools for pneumothorax, nodules or line position
CT tools that flag a pulmonary embolism or brain bleed
Mammography tools that mark suspicious areas

2

Reports and clinical text

Free text, handled by language models

Radiology reports, clinic notes and referral letters hold a lot of information in ordinary sentences. Language models (including LLMs) can pull findings out of reports, draft impressions or flag follow-up recommendations. Old reports are also a cheap source of labels for image models, as you will see below.

FINDINGS:
Heart size is normal. Lungs are clear bilaterally.
No pleural effusion or pneumothorax.
No acute osseous abnormality.

IMPRESSION:
No acute cardiopulmonary process.

Examples

Drafting a report impression from the findings
Finding reports that recommend follow-up imaging
Ambient scribes that turn speech into notes

3

Structured data

Rows and columns from the health record

Age, sex, lab values, vital signs, diagnoses and past exams, stored as a table. Simpler machine learning models often use this kind of data, and it can be combined with images.

Variable	Value	Type
Age	67	Number
Sex	Female	Category
Creatinine	1.4 mg/dL	Number
Prior CT in 90 days	Yes	Yes / no
Reason for exam	Rule out PE	Text

Examples

Suggesting the right scan protocol from the reason for exam
Predicting which patients may miss their appointment

4

Annotations and labels

The answer key that supervised models learn from

An annotation is the "right answer" attached to an example. It can be a single word for the whole image (pneumothorax: yes or no), a box drawn around a finding, or an exact outline of it (a segmentation). More detailed labels take longer to make and teach the model more.

Image needed: box labelChest X-ray with a box drawn around a nodule or pneumothorax.

Image needed: outline labelChest X-ray or CT with the edge of a finding traced (a segmentation).

Examples

Whole-image labels: "pneumothorax present"
Boxes around lung nodules
Outlines of organs for radiation planning

Labels and "ground truth"

For PTX-Detect, every training film needed a label: pneumothorax yes or no. The label is treated as the truth, so it is often called ground truth. But someone had to decide it, and there are several ways to do that:

Four sources of labels: expert read, consensus of several readers, labels mined from old reports, and a reference test such as CT or pathology, each with a trade-off. — Figure 1. Where labels come from. Each source trades accuracy against cost and speed.

Labels can be wrong for many reasons. Radiologists disagree with each other on subtle films. Reports are written for clinicians, not for computers, so software that mines them misreads phrases like "no definite pneumothorax" or "pneumothorax has resolved." And sometimes the true answer is not knowable from the image alone.

If 10% of the labels are wrong, the model learns those mistakes as if they were correct, and the test score is measured against a flawed answer key.

Interactive: label it yourself

You are one of three readers labeling films for PTX-Detect. Read each short description and choose a label. Then see how the other two readers labeled it. There is no right answer here; the point is how often readers differ.

Film AUpright chest X-ray. A clear thin white line runs along the right upper chest with no lung markings beyond it.

Reader 2: pneumothorax. Reader 3: pneumothorax. An obvious case: readers agree, and the label is reliable.

Film BPortable film of a patient lying flat. The left lower chest looks a little darker than the right, with a sharp heart border. No clear line.

Reader 2: pneumothorax (possible "deep sulcus sign," a clue on flat films). Reader 3: no pneumothorax. Subtle films split readers, so whichever label wins is uncertain.

Film CA chest drain is in place. The lung looks fully expanded. The report says "small residual apical pneumothorax, unchanged."

Reader 2: pneumothorax (follows the report). Reader 3: no pneumothorax (cannot see it). Report mining would label it "yes." Is a tiny, treated pneumothorax what we want the model to learn? Lesson 1.5 returns to this.

Image needed

A normal frontal chest X-ray, to sit beside the activity.

Image needed

A chest X-ray with a clear pneumothorax, ideally with the edge of the lung outlined by an expert (shows what a label looks like).

Training, validation and test sets

From Lesson 1.2 you know a model must be checked on data it never trained on. So before training starts, developers split their labeled data into three parts:

A bar split into a training set of about 70%, a validation set of about 15% and a test set of about 15%, compared to practice questions, a mock exam and the real exam. — Figure 2. The labeled data is split before training starts, and each patient goes into only one part.

Training set: the model learns from these, through the training loop.
Validation set: used during development to compare versions and decide when to stop training (the low point in the overfitting figure of Lesson 1.2).
Test set: kept locked away and used once, at the end, to report how good the model is. If developers keep peeking at it and tweaking, it stops being a fair exam.

Interactive: from images to a tested model

These steps are shuffled. Drag them into the order a careful team would follow, then check.

Train the model on the training set
Label each image with the right answer
Run the model once on the locked test set and report the result
Collect images from the hospital archive
Compare versions on the validation set and pick the best one
Split the patients into training, validation and test sets

Leakage and other traps

Data leakage happens when information from the test set sneaks into training, so the test becomes easier than real life. Common forms:

Same patient on both sides. A patient in the ICU may have 15 chest X-rays. If some are in training and some in test, the model can recognize the patient (their ribs, lines and body shape) rather than the disease.
Duplicate images. The same film saved twice, once in each set.
Labels that give the answer away. If every film with a pneumothorax also has a drain in it, the model can learn "drain = pneumothorax." More on this in Lesson 1.5.

Leakage makes a model look better on paper than it will be in practice. When you read about an AI tool, it is fair to ask: was the test set split by patient, and did it come from a different hospital?

Garbage in, garbage out

A model only learns what is in its data. If PTX-Detect was trained only on adults at one city hospital, it has never seen a child, a rural clinic's older scanner, or many patients from groups that hospital rarely serves. It may still give confident answers for them, without having learned anything about them.

So two questions to ask of any dataset: who is in it, and who is missing? Lesson 1.5 shows what happens when a model meets patients unlike the ones it learned from.

Knowledge check

Answer each question to continue. Any answer is fine; read the explanation.

Why is the test set kept separate until the very end?

The test set is the "real exam." If the model or its developers saw it during training, the score would show how well it memorized, not how it will do on new patients.

A team splits 20,000 chest X-rays into training and test sets at random, film by film. Many patients had several films. What is the problem?

Splitting by film instead of by patient lets one person's films land on both sides. The model can then recognize the patient rather than the finding, and the test score comes out too high.

Labels for a model were mined automatically from old radiology reports. What is a likely weakness?

Report mining is fast and cheap, but noisy. Hedged or negative phrases are easy to misread, and the report may describe something not visible on that particular image.

Key takeaways

Radiology AI learns from images, report text, structured health-record data and annotations.
A label is the "right answer" for an example. It comes from expert reads, consensus, old reports or reference tests, and it can be wrong.
Data is split into training, validation and test sets, by patient, before training starts.
Leakage makes a model look better than it is. Missing groups in the data mean the model has never learned about them.

Go further (optional)

Dividing the original dataset (Google ML Crash Course, short page on training/validation/test)
Lesson 2.3 covers medical image data and DICOM in more depth

Next lesson: once PTX-Detect is trained and tested, how do we describe how good it is?