Skip to content
Tutorials Center
My learning

Topics

Lesson 60 of 64 ยท Projects

Project: Passenger Survival Analysis

Analyze a Titanic-style passenger table with pandas: audit missing values, fill ages by group, bin ages, compare survival rates with groupby and pivot tables, and report.

Almost every data analyst has worked through a "ship passenger" dataset: a table of people on a voyage, with columns such as class, sex, age and fare, and one question to answer: who survived, and why? It is a favourite because it is small enough to understand completely, yet messy enough to need real cleaning: missing ages, skewed fares, categories and a yes/no outcome.

This project follows the same path with a synthetic passenger table that we generate ourselves with a fixed random seed. It has the same kinds of columns and the same kinds of problems, but it is our own made-up data, so the numbers are not historical facts. The skills transfer directly when you load a real file from a CSV: read it with pd.read_csv and continue from step 2.

You will practise: building and auditing a table, handling missing values, creating new columns, grouping, pivoting, measuring correlation and writing conclusions that do not claim too much.

flowchart LR
    A["1. Build / load"] --> B["2. Audit<br/>shape, dtypes, missing"]
    B --> C["3. Clean<br/>fill ages, fix types"]
    C --> D["4. Features<br/>age band, family size"]
    D --> E["5. Compare<br/>groupby, pivot"]
    E --> F["6. Check<br/>correlation, counts"]
    F --> G["7. Report"]

A typical analysis path: understand the table before you compare anything.

Step 1: build the data

The generator below creates 900 passengers. Survival depends on sex, class and age through a rule we choose (women and children first, higher classes slightly more likely to survive), plus randomness. Some ages are deleted at random to mimic missing data:

Python Generate the passenger table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Every later example repeats this generator in a short form, because each TryIt runs on its own. In your own project you would keep the table in one DataFrame (or one notebook) and run each step in turn.

Step 2: audit before analysing

First questions for any new table: how big is it, what are the types, how much is missing, and what do the numbers look like?

Python Audit the table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Three findings steer the work: age has gaps (about one passenger in five), fare is right-skewed (the mean is much larger than the median), and survived is stored as 0 and 1, so its mean is the survival rate. That last trick is worth remembering: for a 0/1 column, mean() gives a proportion.

Step 3: clean the missing ages

Dropping all rows with a missing age would throw away a fifth of the passengers and could bias the result. Another option is to fill the gaps with the median age of a similar group. Class and sex are reasonable guides, and groupby(...).transform("median") gives a median for every row, so the fill is one line. Keep a flag so you remember which ages were filled:

Python Fill ages by group and set types

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A filled median is a simple, honest default, but it shrinks the spread of ages. Later you will check that the main conclusions hold even for the passengers whose age was not filled.

Step 4: create features

New columns often reveal patterns that raw columns hide. Two good ones here: an age band (using pd.cut, see Binning) and a family size:

Python Age bands and family size

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

pd.cut assigns each age to a labelled interval; the right edge of every bin is included, so 12.0 falls into "child". family_size counts the passenger plus siblings, spouses, parents and children aboard.

Step 5: compare survival across groups

Now the question itself. The workhorse is groupby on a categorical column, then the mean of the 0/1 outcome, plus the count, because a rate based on very few people means little:

Python Survival rates by sex, class and age band

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The output matches the rule we built into the data: women survive far more often than men in every class, and first class has the best overall rate (the gap between second and third class is small). In a real dataset you do not know the rule. You find it exactly this way, and then ask why before you believe it.

Look at the passenger counts beside the percentages: a cell with 20 passengers can swing by 10 points on one person, while a cell with 300 passengers is steady. Always show n next to a rate.

Step 6: check the story

A good analyst tries to break their own conclusion. Two quick checks:

  1. Does it survive the missing ages? Repeat the sex comparison using only passengers whose age was recorded.
  2. Is a third variable confusing things? Third class is cheaper, and cheaper fares can correlate with class, so check whether fare is a stand-in for class.
Python Robustness checks and correlations

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The sex gap is almost the same when only recorded ages are used, so the missing ages are not driving it. The correlation table shows that fare and pclass are clearly related (correlation -0.54: class 1 is the top class, so a smaller class number goes with a higher fare), which means they are not independent explanations: use one or the other, or compare fares within a class.

Step 7: report

Finish with a short, careful write-up. Something like this:

In this (synthetic) sample of 900 passengers, 53 percent survived. Survival was much higher for women than men in every class, and overall highest in first class. Children had the highest survival of any age band, though the group is small. These patterns hold when passengers with missing ages are excluded. The data cannot show why; it shows association only. Fare largely duplicates class, so it adds little.

Notice what the paragraph does: it states the sample size, reports the main patterns, mentions a robustness check and a limitation, and avoids causal words.

Extension ideas

  • Replace the generator with a real passenger file loaded using pd.read_csv and repeat every step. Compare which of your assumptions about missing data still hold.
  • Do the same exercise on a listings dataset (for example, rental listings with price, neighbourhood and rating): fill missing ratings by neighbourhood, bin prices, and compare averages with groupby and pivot_table. Use medians for prices, because prices are skewed.
  • Do it on a daily case-count dataset: parse dates, use resample and rolling means, and compare regions per 100,000 people.
  • Fit a logistic regression with scikit-learn (see Logistic Regression and Classification) and compare its coefficients with your tables.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does df['survived'].mean() give the survival rate?

    For a 0/1 column, the sum is the number of ones and the mean is that number divided by the count, which is the proportion.
  2. 2. Which line fills missing ages with the median of each class and sex group?

    transform('median') returns a median for every row based on its group, so fillna can use it to fill each gap with the group's median.
  3. 3. Why show the passenger count next to each survival rate?

    One person changes the rate of a small group a lot. Reporting n beside the rate lets readers judge how stable it is.
  4. 4. What does a strong correlation between fare and class tell you about using both as explanations?

    When two columns move together, they overlap. Attributing an effect to one of them separately is unreliable.
  5. 5. Why is it important to repeat the analysis with only recorded ages?

    A robustness check shows whether a conclusion survives when you remove the most questionable data.

Summary

  • A good analysis path: build or load, audit, clean, create features, compare groups, check the story, report.
  • The mean of a 0/1 column is a proportion; groupby(...).agg(rate="mean", n="size") gives rates and counts together.
  • Fill missing values with group medians (groupby().transform("median")) and keep a flag for what you filled.
  • pd.cut creates labelled bands; pivot_table compares two factors at once.
  • Try to break your own conclusion: robustness checks, correlated columns, small groups.
  • Report sample size, patterns, limitations, and avoid causal claims.

Next: Exercises to practise on your own data.