Check a vendor's claim on local data: confusion matrix, sensitivity and specificity, thresholds, ROC curves and subgroups
Step 1 · The question
A pneumothorax detector claims 94% sensitivity. What does it get on local data, and is it equal at both sites?
A pneumothorax is air in the space between the lung and the chest wall, which lets the lung partly collapse. A large one can be life-threatening, so finding it quickly on a chest X-ray matters. Several commercial AI tools read chest X-rays and flag a possible pneumothorax so that the radiologist reads that film first. A vendor's figure such as "94% sensitivity" was measured on the vendor's own test images. A hospital thinking of buying the tool needs to know what it does on its patients, machines and workflow. Checking that is called a local validation, and it is one of the most useful research projects a student can join.
Your table, detector, holds the tool's output on a set of local X-rays: case_id; site (A or B, two hospitals in the same system); truth, which is 1 if a radiologist confirmed a pneumothorax and 0 if not (the reference standard, the best available answer you compare the tool against); and score, the tool's output from 0 to 1. As before, the data is made up and generated from your username.
By the end of this lab you can:
Step 2 · The idea
The tool does not answer yes or no. It gives each X-ray a score between 0 and 1, and someone chooses a threshold: films scoring at or above it are flagged. Every film then lands in one of four boxes, which together make the confusion matrix (also called the 2×2 table). A true positive is a pneumothorax that was flagged; a false negative is one that was missed; a false positive is a flag on a film with no pneumothorax; a true negative is a normal film left unflagged.
Four rates summarize the table. Sensitivity is the share of real pneumothoraces that were flagged; specificity is the share of normal films left alone. Those two describe the tool. The other two describe what a flag means at the bedside: PPV (positive predictive value) is the share of flags that are real, and NPV (negative predictive value) is the share of unflagged films that are truly normal. PPV and NPV change with how common the disease is in the patients being scanned, while sensitivity and specificity do not.
Forty practice X-rays (not your dataset): 12 with a pneumothorax on the top row, 28 without on the bottom, placed by the tool's score. Drag the threshold and watch the table and the four rates change.
No threshold gives perfect sensitivity and perfect specificity at once; lowering it trades false alarms for fewer misses. An ROC curve (receiver operating characteristic) shows that trade at every possible threshold, plotting sensitivity against the false-alarm rate (1 minus specificity). The AUC, the area under that curve, sums it up in one number: 0.5 is no better than a coin toss and 1.0 is perfect. AUC tells you how well the scores rank sick above healthy patients; it does not tell you which threshold to use. That choice depends on the job. A triage tool that only reorders the worklist can afford a low threshold, because a radiologist reads every film anyway.
Finally, a single overall number can hide a site where the tool does much worse, for example a hospital that takes more portable bedside films, uses different X-ray machines or scans sicker patients with tubes and lines in place. Checking performance separately in each group is called a subgroup analysis, and it is how unequal performance is found before patients are harmed.
Step 3 · Warm-up
Two short reading checks. They are not graded and always show the explanation. They also work as a skip test: they cover what the notebook's first parts teach.
What does this code print?
tp, fn, fp, tn = 45, 5, 20, 130 sensitivity = tp / (tp + fn) print(round(sensitivity, 2))
Sensitivity looks only at the patients who have the disease: 45 caught out of 45 + 5 = 50, which is 0.9. 0.87 is the specificity (130 of the 150 without disease left alone), and 0.69 is the PPV (45 of the 65 flags were real). Mixing up the denominators is the most common mistake in this lab, and one of the Fix tasks is built around it.
Drag the lines into an order that works out the sensitivity at a threshold of 0.5.
The prediction column has to be made before you pick out the sick patients, or the smaller table sick would not have a pred column. Among the sick, pred is 1 when the tool flagged the film and 0 when it missed it, so its mean is the share flagged: the sensitivity. Multiplying by 100 gives a percentage, which is how step 5 asks for it.
Step 4 · Notebook
The notebook Lab_3_Evaluating_an_AI_Tool.ipynb loads the tool's scores for your local X-rays. Part 1 builds the confusion matrix at a threshold of 0.5. Part 2 calculates the four rates and has you write sens_spec(), a function you add to your toolkit.py and reuse in Labs 4 and 5. Part 3 has an AI assistant draw the ROC curve, which you then check with a test cell, and asks you to choose a threshold for triage. Part 4 repeats the analysis for each site, and includes subgroup code "written by an AI" that gives a wrong answer for you to fix.
Type the same name into the notebook's first cell:
GITHUB_USERNAME = "your-username"
Colab needs a Google account. It opens the course's copy of the notebook; follow "Save your work to GitHub" below to keep your own.
Tick each part off when you finish it. Your ticks show as progress on the Research Lab map.
Every task in the notebook is tagged with one mode. In Write tasks you write a few lines yourself from an empty cell, with a hint and a hidden solution. In Fill in tasks most of the code is there and you complete the blanks. Reuse tasks import a function you wrote in an earlier lab from your own toolkit.py, the way real projects grow. In Direct the AI tasks you ask an AI assistant (Colab's built-in one is suggested, but any is fine) to write the code, paste it in, run the test cell, and note in the AI-use log what you asked and what you had to change. Fix tasks give you code "written by an AI" with one realistic mistake to find. Read tasks ask you to put lines in order or predict what code will print before you run it.
Early labs lean on writing code yourself, because that is how the basics stick. Later labs lean on reusing your code, directing an AI and checking its work, because that is how research code is written now, and checking is the part a person must still do.
Colab runs the notebook, but it does not keep your work for you. You keep it in your own copy of the course repository on GitHub (a repository, or repo, is a project folder on GitHub with the full history of every saved change). The first time, you make that copy, called a fork; after that, you save into it whenever you finish a part.
github.com/your-username/CA2i-Research-Launchpad.your-username/CA2i-Research-Launchpad, branch main, and keep the file path lesson-notebooks/Lab_3_Evaluating_an_AI_Tool.ipynb. Write a short note about what changed (for example "Part 2 done") and click OK.Extra after this lab: clinical evaluation of AI tools. Which numbers should a department look at before trusting a tool like this one? A companion notebook (about 45 to 60 minutes) follows Metrics Reloaded, an expert framework for choosing validation metrics, and computes balanced accuracy, Expected Cost, AUROC, the Brier score and Net Benefit on your Lab 3 data. Open the extra notebook in Colab or read the guide on the Research Lab page.
Step 5 · Report back
The notebook's last cell prints the numbers below, worked out from your own dataset. Enter them here. If one doesn't match, you get a hint first, then an explanation, and after a third try the answer, so you can find where your analysis went differently. Getting help is fine; the lab is then marked "finished with help".
Loading the answer checker...