Find incidental lung nodules in report text, first with a simple rule, then with a large language model, and check both
Step 1 · The question
How often do reports mention an incidental lung nodule, and can a large language model find them reliably?
An incidental pulmonary nodule is a small round spot in the lung found on a CT done for some other reason, such as a scan for trauma or abdominal pain. Most are harmless, but a few are early lung cancers, so guidelines such as the Fleischner Society's recommend follow-up imaging for many of them, depending on size and the patient's risk. Follow-up is often missed, because the nodule is mentioned in the report but nobody tracks it. Hospitals therefore want to find these mentions in report text automatically.
Radiology reports are unstructured text: free sentences, not neat columns. This lab compares two ways of reading them by computer. One is a simple keyword rule. The other is a large language model (LLM): an AI model trained on huge amounts of text to predict likely next words, which lets it answer questions about a passage. ChatGPT, Gemini and Claude are LLMs. Your table, reports, holds about 120 short made-up chest CT reports, generated from your username, with report_id, text, truth_nodule (1 if a person reading the report would say it describes a lung nodule, the reference standard) and llm_nodule (an LLM's saved answer to the same question). Because the LLM answers are saved, you need no account or key; an optional cell shows how to ask Colab's built-in Gemini or another model yourself.
By the end of this lab you can:
sens_spec().Step 2 · The idea
The obvious rule is: flag a report if it contains the word "nodule". It is easy to write, fast, and gives the same answer every time. But radiology reports are full of negation ("No pulmonary nodule"), synonyms ("opacity", "lesion") and history ("previously described nodule, now resolved"), and a keyword rule cannot tell any of these apart. An LLM reads the whole sentence, so it handles most of them, but it makes errors of its own, and its errors are harder to spot because the answer always sounds reasonable.
Nine report excerpts (practice data). For each one, decide whether it describes a lung nodule. Then run the keyword rule and show the saved LLM answers, and see where each goes wrong.
On your 120 reports, the notebook measures both methods against the reference labels with the sens_spec() function you wrote in Lab 3. A false positive here is a report flagged without a nodule; a false negative is a nodule the method missed, which is the error that costs a patient their follow-up.
Checking LLM output is the core skill. Compare it with a sample of reports a person has labeled, read every disagreement, and remember that the same question can get a different answer if asked again or worded differently. One rule comes before all of these: real patient reports contain protected health information and must never be pasted into a public chatbot. Research on real reports uses models approved by the hospital, under its privacy rules.
Step 3 · Warm-up
Two short reading checks. They are not graded and always show the explanation. They also work as a skip test: they cover what the notebook's first parts teach.
What does this code print?
text = "No pulmonary nodule. Lungs clear."
print("nodule" in text)
in checks whether the letters "nodule" appear anywhere in the text, and they do. Python has no idea that "No" changes the meaning. That is the negation trap: a keyword rule flags this normal report as positive.
Drag the lines into the order that applies the keyword rule and counts the reports where it disagrees with the reference.
str.contains applies the rule to every report at once, and astype(int) turns True and False into 1 and 0 so they can be compared with the reference column. The rows where the two differ are the rule's errors, and reading them one by one is how you find out why it fails.
Step 4 · Notebook
The notebook Lab_5_Reports_and_LLMs.ipynb loads your reports. Part 1 explores them as text: how long they are, which words they use, and how often the reference labels say a nodule is present (the first half of the lab's question). Part 2 builds the keyword rule, improves it to handle negation, and compares it with the saved LLM answers using your sens_spec(). Part 3 checks the LLM properly: you read its errors, sort them by type, and fix AI-written checking code with a planted bug. An optional cell shows how to ask a live model yourself.
Type the same name into the notebook's first cell:
GITHUB_USERNAME = "your-username"
Colab needs a Google account. It opens the course's copy of the notebook; follow "Save your work to GitHub" below to keep your own.
Tick each part off when you finish it. Your ticks show as progress on the Research Lab map.
Every task in the notebook is tagged with one mode. In Write tasks you write a few lines yourself from an empty cell, with a hint and a hidden solution. In Fill in tasks most of the code is there and you complete the blanks. Reuse tasks import a function you wrote in an earlier lab from your own toolkit.py, the way real projects grow. In Direct the AI tasks you ask an AI assistant (Colab's built-in one is suggested, but any is fine) to write the code, paste it in, run the test cell, and note in the AI-use log what you asked and what you had to change. Fix tasks give you code "written by an AI" with one realistic mistake to find. Read tasks ask you to put lines in order or predict what code will print before you run it.
Early labs lean on writing code yourself, because that is how the basics stick. Later labs lean on reusing your code, directing an AI and checking its work, because that is how research code is written now, and checking is the part a person must still do.
Colab runs the notebook, but it does not keep your work for you. You keep it in your own copy of the course repository on GitHub (a repository, or repo, is a project folder on GitHub with the full history of every saved change). The first time, you make that copy, called a fork; after that, you save into it whenever you finish a part.
github.com/your-username/CA2i-Research-Launchpad.your-username/CA2i-Research-Launchpad, branch main, and keep the file path lesson-notebooks/Lab_5_Reports_and_LLMs.ipynb. Write a short note about what changed (for example "Part 2 done") and click OK.Step 5 · Report back
The notebook's last cell prints the numbers below, worked out from your own dataset. Enter them here. If one doesn't match, you get a hint first, then an explanation, and after a third try the answer, so you can find where your analysis went differently. Getting help is fine; the lab is then marked "finished with help".
Loading the answer checker...