Skip to content
Tutorials Center
My learning

Topics

Lesson 41 of 77 ยท Databases and Data Interview Questions

Data Science and pandas Interview Questions

Prepare for data interviews: pandas groupby, merge types, missing data, bias vs variance and evaluation metrics like precision and recall, with runnable code.

Data roles mix two kinds of questions. The first kind tests whether you can wrangle a table with pandas without writing loops. The second kind tests whether you understand what a model's numbers mean: why it overfits, and why 99% accuracy can still be useless. This lesson covers both with short, runnable examples.

If a topic feels shaky, read the full lessons first: groupby, merge and join and missing data.

What data interviews check

Wrangling groupby, merge, reshape.
Messy data Missing values and duplicates.
Model sense Bias, variance, overfitting.
Metrics Pick the one that fits the cost.
1 / 2

groupby: split, apply, combine

Q: What does groupby do? It splits rows into groups by one or more keys, applies a function to each group, and combines the results. Say the three words "split, apply, combine" and you have the core answer.

Follow-ups usually ask for the difference between agg, transform and apply. agg returns one row per group. transform returns a result with the same length as the input, so you can add it as a column (for example, each row's share of its group total). apply is the flexible fallback that runs any function on each group, but it is the slowest.

Python agg versus transform

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that agg shrinks five rows to two, while transform keeps all five. Use transform whenever you need group statistics next to the original rows.

merge types

Q: Explain inner, left, right and outer joins. An inner join keeps only keys present in both tables. A left join keeps every row of the left table and fills the right side with NaN when there is no match. Right is the mirror image. An outer join keeps every key from both. The usual trap is a many-to-many key: if both sides repeat a key, the result multiplies rows. Pass validate="one_to_one" or "many_to_one" to make pandas raise an error instead of silently multiplying.

Python Join types and the row-count trap

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

If the interviewer asks how this maps to SQL, it is the same idea as the joins in SQL: Query-Writing Interview Questions.

Missing data

Q: How do you handle missing values? There is no single answer; the right one depends on why the data is missing. Start by measuring: df.isna().sum(). Then choose:

  • Drop rows or columns (dropna) when few are affected and missing values look random.
  • Fill with a constant, the median (robust to outliers) or a group-wise value (groupby(...).transform("median")).
  • Flag the gap with an indicator column when "missing" itself carries information.
  • Model-based imputation when accuracy matters and you can afford it.

Always fit the fill value on training data only, then reuse it on test data, otherwise information leaks from test into training.

Python Group-wise median fill and a missing flag

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last line is a favorite: NaN never equals itself, so use isna, not ==.

Bias versus variance

Q: What is the bias-variance tradeoff? Bias is error from a model that is too simple to capture the pattern (underfitting). Variance is error from a model so flexible that it chases noise in the training set (overfitting). Making a model more complex lowers bias but raises variance. You want the complexity where validation error is lowest.

The classic symptom check: high training error and high validation error means high bias; low training error but high validation error means high variance. Fixes differ. For high variance: more data, regularization, simpler model, early stopping. For high bias: more features or a more expressive model.

Python Underfit, good fit and overfit with polynomials

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Training error always falls as degree rises, but validation error does not. That gap is the picture to describe on a whiteboard.

Evaluation metrics

Q: Why is accuracy not enough? With imbalanced classes, a model that always predicts the majority class scores high accuracy and finds nothing. Use the confusion matrix counts: true positives (TP), false positives (FP), false negatives (FN) and true negatives (TN).

  • Precision = TP / (TP + FP): of the items flagged, how many were right?
  • Recall = TP / (TP + FN): of the real positives, how many did we find?
  • F1 = harmonic mean of precision and recall.

Pick by cost. Spam filter: a false positive (good mail lost) is costly, so favor precision. Cancer screening: a miss is costly, so favor recall.

Python Accuracy hides a useless model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The lazy model has 95% accuracy and zero recall. For a deeper tour, see Data Science and Machine Learning.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which pandas method returns a result with the same length as the input, so it can be added back as a column?

    transform broadcasts the group result back to every row; agg returns one row per group.
  2. 2. You merge two tables and the row count suddenly grows. What is the most likely cause?

    Each matching pair of duplicated keys produces a row. validate= makes pandas raise an error.
  3. 3. Training error is very low but validation error is high. What is the problem?

    A big gap between training and validation error signals the model memorized noise.
  4. 4. A fraud model must catch as many frauds as possible and a few false alarms are acceptable. Which metric do you prioritize?

    Recall measures the share of real positives found, which is what the cost of a miss demands.

Summary

  • groupby is split, apply, combine; transform keeps the input length, agg shrinks it.
  • Know the four join types, and guard against many-to-many row explosions with validate.
  • Missing data: measure first, then drop, fill (ideally by group), or flag; fit fills on training data only.
  • Bias is underfitting, variance is overfitting; compare training and validation error.
  • Accuracy misleads on imbalanced data; choose precision or recall by the cost of each error.

Next: Complexity and Data Structure Questions.