Clean a messy patient table, describe it in a Table 1, and test a difference between two groups
Step 1 · The question
Are patients sent for CT older than those sent for MRI?
It sounds like a two-line answer: work out the average age in each group and compare. In practice, data exported from a hospital system is messy. The same thing is written in different ways, values are missing, some are impossible, and some patients appear twice. Averages computed on messy data can be badly wrong, and nothing on the screen warns you. So this lab follows the order every clinical study follows: clean the data by stated rules, describe who is in it, look at it in a chart, and only then test the difference.
Your table, clinical, lists patients sent for CT or MRI with the columns patient_id, modality, age and sex. It has been made messy on purpose, in the ways real exports are: sex recorded as M, male, F or f; missing ages; a few impossible ages such as 999; and duplicate rows. The notebook states the cleaning rules, so everyone cleans the same way. As in Lab 1, the table is made up and generated from your GitHub username.
By the end of this lab you can:
Step 2 · The idea
Data cleaning means finding and dealing with values that are wrong, missing or inconsistent before any analysis. Some rows are kept and corrected (writing "male" as "M" so it is counted with the other M rows); others are removed (a duplicate row, or an age of 999 that was really a code for "unknown"). Good practice is to write the rules down before you start and to report how many rows each rule removed, so a reader can judge whether the cleaning could have changed the answer.
Twelve rows from an export (practice data). Click every row that needs work, either removing or correcting, then check your picks. Watch what the messy rows do to the average ages.
One impossible age turned the comparison upside down. That is why cleaning comes first, and why it is written into the notebook as code: anyone can rerun it and get the same rows.
Next comes Table 1, so called because it is the first table in nearly every clinical paper. It describes who was studied, group by group: how many patients, their age as a mean (the average) with a standard deviation (how spread out the values are), or as a median (the middle value) with an interquartile range when the values are skewed, and counts with percentages for categories such as sex. A reader looks at Table 1 to decide whether your patients resemble theirs.
Finally, even if the CT patients turn out a few years older on average, a difference that size could appear by chance in a small sample. A statistical test asks how surprising your difference would be if there were really no difference between the groups. Its answer is the p-value: a small p-value (by convention, below 0.05) means a difference this large would be unusual by chance alone. It does not tell you the difference is large, or that it matters clinically. For comparing two means, the notebook uses Welch's t-test.
Step 3 · Warm-up
Two short reading checks. They are not graded and always show the explanation. They also work as a skip test: they cover what the notebook's first parts teach.
What does this code print?
df = pd.DataFrame({"age": [70, 80, None, 999]})
df = df[df["age"] < 120]
print(len(df))
The middle line keeps only the rows where the condition is true. 999 fails it. The missing value (pandas shows it as NaN, "not a number") also fails, because any comparison with a missing value counts as false. That is handy here, but in the notebook you drop missing values on purpose with dropna(), so that you can count and report how many there were.
Drag the lines into an order that gives the right mean age for each modality.
Cleaning has to happen before the summary: if the duplicates are still there when the means are worked out, a patient listed twice counts twice. groupby("modality") splits the table into a CT group and an MRI group, and ["age"].mean() averages the age column within each group. round(1) keeps one decimal place, which is how you report the numbers in step 5.
Step 4 · Notebook
The notebook Lab_2_Clinical_Data.ipynb loads your messy table and works through the four steps above. Part 1 applies the written cleaning rules and keeps a tally of what each rule removed. Part 2 builds Table 1 for the CT and MRI groups. Part 3 draws the age distribution of each group so you can see the difference before testing it. Part 4 runs the tests, Welch's t-test for age and a chi-square test for sex, and asks you to write one sentence interpreting each. Some tasks reuse the loading function you put in toolkit.py in Lab 1.
Type the same name into the notebook's first cell:
GITHUB_USERNAME = "your-username"
Colab needs a Google account. It opens the course's copy of the notebook; follow "Save your work to GitHub" below to keep your own.
Tick each part off when you finish it. Your ticks show as progress on the Research Lab map.
Every task in the notebook is tagged with one mode. In Write tasks you write a few lines yourself from an empty cell, with a hint and a hidden solution. In Fill in tasks most of the code is there and you complete the blanks. Reuse tasks import a function you wrote in an earlier lab from your own toolkit.py, the way real projects grow. In Direct the AI tasks you ask an AI assistant (Colab's built-in one is suggested, but any is fine) to write the code, paste it in, run the test cell, and note in the AI-use log what you asked and what you had to change. Fix tasks give you code "written by an AI" with one realistic mistake to find. Read tasks ask you to put lines in order or predict what code will print before you run it.
Early labs lean on writing code yourself, because that is how the basics stick. Later labs lean on reusing your code, directing an AI and checking its work, because that is how research code is written now, and checking is the part a person must still do.
Colab runs the notebook, but it does not keep your work for you. You keep it in your own copy of the course repository on GitHub (a repository, or repo, is a project folder on GitHub with the full history of every saved change). The first time, you make that copy, called a fork; after that, you save into it whenever you finish a part.
github.com/your-username/CA2i-Research-Launchpad.your-username/CA2i-Research-Launchpad, branch main, and keep the file path lesson-notebooks/Lab_2_Clinical_Data.ipynb. Write a short note about what changed (for example "Part 2 done") and click OK.Step 5 · Report back
The notebook's last cell prints the numbers below, worked out from your own dataset. Enter them here. If one doesn't match, you get a hint first, then an explanation, and after a third try the answer, so you can find where your analysis went differently. Getting help is fine; the lab is then marked "finished with help".
Loading the answer checker...