Research Lab · optional track
Data science for clinical AI research
Six hands-on labs, from your first line of Python to a project of your own. Each one starts with a real clinical question. You work in Google Colab (free notebooks that run in your browser) and save to GitHub (a site that keeps every version of your code), so your progress becomes a portfolio.
The labs
One path, six stops
Work through them in order. Each lab opens with a short warm-up; if it's easy, you can skim ahead. Lab 6, your own project, is optional. Every lab page is open to read at any time.
Clinical evaluation of AI tools
Choosing the right numbers
A vendor's headline accuracy rarely tells a department what it needs to know. This guide follows Metrics Reloaded, a framework built by an international group of image analysis and clinical experts to help researchers choose validation metrics (the numbers used to judge a model) that fit the clinical problem.
Suppose a pneumothorax detector is tested on 1,000 chest X-rays and 20 of them show a pneumothorax. A model that calls every image normal is right 980 times: 98% accuracy, while missing every patient who needed a chest drain. The number was correct; it was the wrong number for the question.
The Metrics Reloaded authors found that problems like this are common in published image analysis research: the metrics chosen often don't match what matters clinically, which makes it hard to know whether a tool is really better and slows its safe use in patients. Their answer is to choose metrics from a description of the problem, not from habit.
Lab 3 teaches the basic tools (the confusion matrix, sensitivity, specificity and ROC curves). This guide and its notebook show how to decide which of them, and which others, belong in a validation report.
- Describe the problem: the "problem fingerprint". First name the problem category. Metrics Reloaded covers four: image-level classification (one answer per image, such as "pneumothorax or not"), object detection (finding and locating each lesion), semantic segmentation (labeling every pixel, such as outlining the lungs) and instance segmentation (outlining each separate object, such as each nodule). Then record what matters for metric choice: what the clinician cares about, how common each class is (prevalence), how small the targets are, what the model outputs (a score or a yes/no), whether a decision threshold has been set, and whether its scores need to be trustworthy probabilities (calibration).
- Select metrics. Follow the path for your category. Each answer in the fingerprint narrows the choice, and the framework warns about known pitfalls along the way.
- Apply them. Compute the chosen metrics on your data, report them together, and look at them per class and per subgroup (for example, per hospital site).
The free Metrics Reloaded online tool walks you through these steps with questions and suggests a set of metrics at the end.
For image-level classification, the category most radiology AI tools fall into, the recommendations come in layers:
One overall count-based metric
Expected Cost when mistakes carry different costs, as in medicine, where a missed pneumothorax is worse than a false alarm. Otherwise accuracy, balanced accuracy (the average of sensitivity and specificity) or MCC (Matthews correlation coefficient), chosen by how class prevalence in your test set compares with the patients the tool will see.
Per-class metrics, optional
When a threshold turns scores into yes/no answers, a per-class metric such as the F-beta score (which combines sensitivity with positive predictive value) shows how each class fares on its own.
A threshold-free metric, always
AUROC (area under the ROC curve) is the default because it doesn't depend on prevalence. Average precision is an alternative in some class-balance settings.
Calibration, if it matters
If clinicians will read the score as a probability ("82% chance of pneumothorax"), add a calibration metric such as the Brier score. The paper also highlights Net Benefit, which weighs correct detections against unnecessary follow-up, for decisions based on a cost-benefit rule.
The companion notebook takes the pneumothorax detector from Lab 3, fills in its problem fingerprint, shows why accuracy misleads, and computes balanced accuracy, MCC, Expected Cost, AUROC, average precision, the Brier score and Net Benefit on your own data. It takes about 45 to 60 minutes; do it after Lab 3.
Sources
- Maier-Hein L, Reinke A, Godau P, et al. Metrics reloaded: recommendations for image analysis validation. Nat Methods. 2024;21(2):195-212. doi:10.1038/s41592-023-02151-z. PubMed 38347141.
- Reinke A, Tizabi MD, Baumgartner M, et al. Understanding metric-related pitfalls in image analysis validation. Nat Methods. 2024;21(2):182-194. doi:10.1038/s41592-023-02150-0.
- Metrics Reloaded online tool: metrics-reloaded.dkfz.de. Python package: Project-MONAI/MetricsReloaded.
How you'll work with code
Less typing from scratch, more checking
Every notebook task is one of six kinds. Early labs build fluency by having you write code yourself. Later labs work the way research code is written now: you reuse your own earlier functions, have an AI assistant draft code, and check it with a test.
Where do I start?
Four quick questions
Not graded and not saved. The warm-up at the start of each lab will confirm it.
Getting involved
From skills to a research position
The labs give you skills and a portfolio. These notes cover the next step: joining a real project.
Roles students fill
What each job on a project involves and which lab prepares you.
Data cleaning and chart review. Pulling ages, diagnoses and outcomes from records into a tidy table, and fixing what is missing or wrong. Lab 2.
Analysis. Computing sensitivity, specificity and confidence intervals, comparing sites and subgroups. Lab 3.
Image work. Organizing image sets, labeling findings, running and checking a model. Lab 4.
Text and reports. Finding cases in report text with rules or a large language model, and checking the results. Lab 5.
Literature and writing. Summarizing what is already known and drafting the abstract. Lab 6.
Finding a mentor
Who to ask, a short email template, and what to attach.
Look for radiology faculty who have published on AI or imaging informatics in the last few years, and for data science or biomedical informatics faculty who work with imaging. Your school's research office or a radiology residency program director can often point you to them.
Keep the first email short and specific:
Subject: Medical student interested in your imaging AI work Dear Dr. [Name], I am a [year] medical student at [school]. I read your paper on [topic] and was interested in [one specific point]. I have been learning Python and data analysis through the AI2Practice Research Lab; my work is at [GitHub link]. Would you have 15 minutes to talk about whether there is a way I could help on a current project, for example with data cleaning or analysis? Thank you, [Name]
Attach or link one thing you made, such as your Lab 6 repository and abstract. If you don't hear back in two weeks, one polite follow-up is normal.
Conferences
Where students present imaging AI work.
AUR (Association of University Radiologists) annual meeting: education and research, with trainee sessions.
RSNA annual meeting: the largest radiology meeting, with medical student and trainee abstracts.
SIIM (Society for Imaging Informatics in Medicine): informatics and AI in practice.
MIDL (Medical Imaging with Deep Learning): a technical machine learning meeting.
Abstract deadlines often fall several months before each meeting and change every year, so check each site early.
Open datasets
What's inside and how to get access.
MedMNIST: small, ready-to-use image sets (chest X-ray, pathology, CT and more). Free download; used in Lab 4.
CheXpert: over 200,000 chest X-rays with report-derived labels. Free after registration.
MIMIC-CXR: chest X-rays with their full reports. Needs a PhysioNet account and short human-research training.
The Cancer Imaging Archive: CT, MRI and PET collections, many with clinical data.
Challenges
Competitions worth trying after Lab 4.
RSNA AI challenges: a new imaging task most years, with public data and a leaderboard.
Kaggle: hosts the RSNA challenges and many other medical tasks; past competitions keep their data and shared notebooks, which are good to learn from.
Grand Challenge: academic medical imaging challenges, often tied to MICCAI and other conferences.
Joining as a team with a resident or graduate student is a good way to learn.
Funding
Student grants and summer programs.
The RSNA Research & Education Foundation offers medical student research grants, usually with a faculty mentor.
Most medical schools run summer research programs, often funded by NIH short-term training grants. Ask your research office what is open to first- and second-year students.
Specialty societies and some departments fund student travel to present an accepted abstract; ask before you submit.
Programs, links and deadlines change; check each site for the current details.