Overview
PTX-Detect scored 94% sensitivity in testing. This lesson is about why that number may not hold when the tool reaches your hospital, the surprising things models sometimes learn instead of the disease, and the habits that keep you safe when you work alongside AI. Each failure here has happened with real, published models. About 15-20 minutes.
By the end of this lesson you can:
- Explain why a model that works at one hospital can fail at another (dataset shift).
- Describe shortcut learning with a real example.
- Recognize overconfidence and "hallucination" in AI output.
- Name three habits for using AI safely.
New hospital, new problems
A model learns from the images it was given. If the images at a new hospital look different, the model is working outside what it learned. This is called dataset shift. Images can differ because of the scanner brand and settings, the patients (age, illness, body size), how films are taken (upright in the department or lying flat in the ICU), and even the software that processes the image.
Remember from Lesson 1.2 that a model is only checked properly on data it never trained on. A test set from a different hospital (an "external" test) is a much tougher and more honest check than one from the same hospital.
Shortcuts: right answer, wrong reason
Training rewards the model for getting the label right, and nothing else. If some feature of the image happens to go along with the label, the model will happily use it, even if it has nothing to do with the disease. This is called shortcut learning. Two real examples:
- The hospital as a clue. A pneumonia model trained on chest X-rays from several hospitals learned to tell which hospital a film came from. Pneumonia was much more common at some hospitals than others, so "which hospital" became a clue for "pneumonia." Its performance dropped on films from a hospital it hadn't seen. (Zech et al., PLoS Medicine 2018)
- Text and position instead of lungs. Early COVID-19 chest X-ray models were trained with COVID-positive and negative films that came from different sources. The models leaned on things like text markers and patient positioning, and areas outside the lungs, rather than on lung findings. Tested on new data, they did much worse. (DeGrave et al., Nature Machine Intelligence 2021)
Shortcuts are hard to catch because the test score looks fine whenever the test set shares the same shortcut as the training set.
Try it: spot the shortcut
This drawing shows a portable chest X-ray that PTX-Detect flagged for pneumothorax. Click each numbered region to find out whether it is the real finding or something the model might have used as a shortcut.
Where did PTX-Detect look?
Click a numbered region on the image. Some regions are the actual finding; others are clues that have nothing to do with the lung.
1. The "PORTABLE" text marker
Portable films are taken on patients too sick to go to the X-ray department, and sicker patients have more pneumothoraces. A model can learn "PORTABLE means more likely positive" without looking at the lungs at all. This is the same kind of clue as the hospital in the Zech study.
2. The chest drain
A chest drain is the treatment for a pneumothorax. If many positive training films already had a drain in place, the model can learn "tube means pneumothorax." It then does well on patients who are already treated and poorly on the ones who still need help. See "Hidden stratification" below.
3. The edge of the collapsed lung
The thin white line near the top of the right lung, with no lung markings beyond it, is the visceral pleural line: the edge of the partly collapsed lung. This is what a radiologist looks for, and what we hope the model learned.
4. The side marker and image edge
Side markers, text and the edges of the image differ between hospitals, machines and technologists. The COVID-19 models in the DeGrave study leaned on regions like these, outside the lungs, because they happened to differ between positive and negative films.
A real portable chest X-ray with a visible text marker (e.g. "PORTABLE" or "SUPINE"), to replace or sit beside the drawing.
A chest X-ray showing a pneumothorax with a chest drain in place.
Try it: will it transfer?
PTX-Detect was trained and tested on upright chest X-rays of adult outpatients at one large hospital, using one brand of X-ray machine. For each new setting, decide whether you would expect its published performance to carry over.
Callback: go back to the model you trained in Teachable Machine in Lesson 1.2. Change the lighting, move to another room, or hold the pen at an odd angle. How quickly does it fail? That is dataset shift on your own desk.
Confident and wrong
- Confidence scores are not guarantees. A score of 0.97 means the film looks a lot like the positives the model trained on, not that it is 97% certain to be right for this patient. Under dataset shift, models are often confidently wrong.
- Heatmaps can mislead. Many tools show a colored overlay of "where the model looked." These maps are rough, and a map that lands on the right spot does not prove the model used the right reason. A map on the wrong spot, though, is a useful warning sign.
- Generative AI can make things up. Chatbots and report-drafting tools can produce fluent, convincing text that is false, such as a finding that isn't there or a reference that doesn't exist. This is called hallucination. Always check generated text against the source.
An AI heatmap (colored overlay) on a chest X-ray, ideally one that highlights the wrong area, such as a tube or text marker. Could come from a published paper figure if its license allows.
What you can do
You don't need to be a computer scientist to use AI safely. Three habits go a long way:
- Look first, then check the AI. Form your own read before you see the AI's answer, so the tool checks you rather than replaces you. Lesson 3.3 covers why this matters for learning.
- Ask what it learned from. Which patients, which hospitals, which machines, and how was it tested? If your patient is unlike the training data, trust the output less. Lesson 3.2 shows how to read the evidence.
- Speak up about errors. When the AI and your read disagree, look again carefully and ask a senior colleague. If the AI was wrong, report it. Errors that nobody reports never get fixed.
Knowledge check
Answer each question to continue. Any answer is fine; read the explanation.
A fracture-detection model scores very well at the hospital where it was built. At a second hospital with a different X-ray vendor, it misses many more fractures. What is the most likely explanation?
Different machines produce images that look different to the model. Its performance was measured on images like its training data; the new images are outside that. Models don't forget once trained.
A heatmap shows a model focused on the correct area of the lung. What can you conclude?
A heatmap is a rough picture of which pixels influenced the answer. A well-placed map is reassuring but not proof. A map on a tube or text marker, however, is a red flag worth noticing.
On a rotation, the AI says a chest X-ray is normal, but you think you see a small pneumothorax. What should you do?
Disagreement is exactly when a second look matters. The AI can miss findings, especially subtle ones or in patients unlike its training data. Raise it, learn from the answer, and report the AI's error if it was wrong.
Key takeaways
- A published score describes the hospital and patients where the model was tested. New scanners, patients or techniques cause dataset shift.
- Models can learn shortcuts, such as text markers, tubes or the hospital itself, instead of the disease.
- A good overall score can hide a group the model fails on (hidden stratification).
- Confidence scores and heatmaps are not proof; generative AI can hallucinate.
- Look first, ask what it learned from, and speak up about errors.
Module 1 complete
You now know what AI is, how it learns, what it learns from, how it is measured and how it fails. Module 2 covers radiology itself: how images are made, what they are to a computer, how a scan moves through the hospital and how to read the report, and Module 3 covers judging the evidence and using AI without losing your own skills.
Go further (optional)
- Zech et al. 2018 pneumonia model and hospital site
- DeGrave et al. 2021 COVID-19 shortcuts
- Oakden-Rayner et al. 2020 hidden stratification