Skip to content
Tutorials Center
My learning

Topics

Lesson 52 of 64 ยท Quality and Production

Common pandas Pitfalls

The pandas surprises that bite everyone: chained assignment, index alignment, object dtype, integer NaN, dropped groupby keys, inplace myths and float precision.

Most pandas bugs are not crashes. They are quiet: a column that was never updated, a sum that is NaN because two indexes did not line up, a group that vanished from a report. The code runs, the numbers look plausible, and the mistake travels downstream.

This lesson collects the surprises that cause the most quiet bugs, shows each one on a tiny example, and gives you the habit that prevents it. Everything here is for pandas 3, where Copy-on-Write is always on and several old traps have changed shape. Where an older tutorial tells you something different, this page is the one to trust.

Pitfall 1: chained assignment never works

Selecting rows and then assigning into the result looks natural:

Python
df[df["score"] > 55]["score"] = 0     # does NOT change df

That is two operations. df[mask] builds a new object, and ["score"] = 0 modifies that temporary object, which is then thrown away. In older pandas this sometimes worked and sometimes did not, which is where the famous SettingWithCopyWarning came from. In pandas 3, Copy-on-Write removes the guessing: every result of an indexing operation behaves as its own copy, so a chained assignment can never update the original. pandas now emits a ChainedAssignmentError warning to tell you.

flowchart LR
    A["df"] --> B["df[mask]<br/>(new temporary object)"]
    B --> C["['score'] = 0<br/>(changes the temporary)"]
    C --> D["temporary is discarded"]
    A -. "unchanged" .-> E["df still has old values"]

A chained assignment edits a temporary copy, so the original DataFrame is never touched.

The fix is one single step with .loc, which selects rows and columns together and writes into the original:

Python Chained assignment versus .loc

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The same rule explains why view = df["score"] followed by view[0] = 99 no longer changes df. If you want to change a frame, assign into the frame itself (with .loc, df["col"] = ... or assign), never into an intermediate object.

Pitfall 2: index alignment surprises

Arithmetic between two Series or DataFrames matches values by label, not by position. That is a feature (it keeps data attached to the right entity) and a trap when the labels differ:

  • Labels present in only one side produce NaN in the result.
  • Assigning a Series to a column aligns on the index, so a shuffled Series lands in different rows than you expect.
  • Duplicate labels make alignment ambiguous and sometimes raise.
flowchart TB
    subgraph L["left index"]
        l1["x: 1"]
        l2["y: 2"]
        l3["z: 3"]
    end
    subgraph R["right index"]
        r1["y: 10"]
        r2["z: 20"]
        r3["w: 30"]
    end
    L --> O["sum aligned by label:<br/>w NaN, x NaN, y 12, z 23"]
    R --> O

Only labels found on both sides get a number; every other label becomes NaN.

Python Alignment by label

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When you really do mean "line these up by position", pass .to_numpy() or call reset_index(drop=True) on both sides first. When you mean "line them up by label", check the index before you trust the result. The Index lesson goes deeper.

Pitfall 3: object dtype and mixed-type columns

A column that holds numbers and text together cannot be stored in a fast numeric array. pandas falls back to object, where every cell is an arbitrary Python object. Sums fail or concatenate, sorting compares strings, and everything is slower.

The classic source is a CSV column with a stray "n/a" or "1,200". Numbers that arrive as text sort alphabetically: "100" sorts before "9". Fix the column once, early, with pd.to_numeric, and decide what to do with the bad cells (errors="coerce" turns them into NaN, which you can then count).

Python Mixed types and text numbers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In pandas 3, columns of text use the dedicated str dtype by default, which is faster and stricter than object. A column of genuinely mixed values still ends up as object, which is your cue that the data needs cleaning. See Data Types and Conversion.

Pitfall 4: missing values change dtypes

Plain NumPy integers cannot hold "missing". The moment a NaN appears in an integer column (through reindex, a merge, or a read), the whole column becomes float64, and 3 becomes 3.0. IDs that turn into 3.0 and then fail to match a text "3" are a standard source of broken joins.

Two defences:

  • Use a nullable integer dtype such as "Int64" (capital I) when integers can be missing. It stores <NA> and stays integer.
  • Never test for missing with == np.nan. NaN is not equal to anything, including itself, so use isna().

pandas 3 is also strict about writing a value a column cannot hold. Putting 1.5 into an int64 column with .loc raises a TypeError instead of silently converting; cast the column first with astype.

Python Integer columns and missing values

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

More on this in Missing Data.

Pitfall 5: groupby silently drops missing keys

By default groupby leaves out rows whose key is missing. A report grouped by region quietly loses every row with an unknown region, and the totals no longer add up to the grand total. Always compare the sum of the groups with the sum of the whole, or pass dropna=False to keep an explicit NaN group.

Python
g = pd.DataFrame({"region": ["N", None, "N", "S"], "sales": [1, 2, 3, 4]})
g.groupby("region")["sales"].sum()                 # N 4, S 4  (total 8, not 10)
g.groupby("region", dropna=False)["sales"].sum()   # also shows the NaN group

groupby Basics covers the other options. A cheap safety net is a check that the pieces add up, which you will build properly in Testing and Validating DataFrames.

Pitfall 6: inplace does not do what you think

Many methods accept inplace=True, and it is tempting to read it as "faster, no copy". It is neither. In current pandas it mostly makes a copy internally and swaps the result in, saves no memory, and returns None, which breaks method chaining and produces the notorious df = df.sort_values(..., inplace=True) bug that sets df to None. The pandas team has been moving away from it.

Prefer returning new objects and assigning the result:

Python
df = df.sort_values("score")          # clear, chainable
df = df.drop_duplicates()

Hidden loops are a close cousin of this problem. Changing the row you get from for _, row in df.iterrows() changes a copy, not the frame:

Python inplace returns None, iterrows edits a copy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Pitfall 7: float precision and rounding

Floats are binary approximations. 0.1 + 0.2 is not exactly 0.3, and equality checks on computed floats fail. Prices multiplied by quantities pick up tiny errors, and totals compared with == disagree. Two habits fix almost everything:

  • Compare with a tolerance: np.isclose(a, b) for values, and the rtol/atol arguments of the testing helpers for frames.
  • For money, store whole cents as integers, or round once at the end, not at every step.

Also remember that round() follows round half to even (banker's rounding): 0.5 rounds to 0, 1.5 to 2 and 2.5 to 2.

Python Floats are approximations

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

More traps worth knowing

  • Attribute access. df.count is the method, not a column called count. Use df["count"].
  • Duplicate index labels. s.loc["a"] returns several rows when the label repeats, and later joins multiply rows. Check df.index.is_unique.
  • and / or in filters. Use &, | and ~ with parentheses around each comparison. See Filtering with Conditions.
  • Reading dates as text. A date column that stayed str sorts and groups wrongly. Convert with pd.to_datetime.
  • Appending in a loop. Growing a frame row by row is slow; collect rows in a list and build once.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In pandas 3, what happens when you run df[df['score'] > 55]['score'] = 0?

    Under Copy-on-Write every indexing result acts as a copy. Chained assignment edits that temporary object, so the original stays as it was. Use df.loc[mask, 'score'] = 0 instead.
  2. 2. You add two Series with different index labels. What do you get for a label that exists in only one of them?

    Arithmetic aligns by label. A missing partner gives NaN unless you use a method such as add(other, fill_value=0).
  3. 3. Why does an integer ID column often turn into floats after a merge or reindex?

    The classic int64 dtype has no missing marker. Use a nullable dtype such as Int64 when integers can be missing.
  4. 4. A report grouped by region totals less than the whole table. What is a likely cause?

    Rows with a missing group key are excluded by default. Pass dropna=False or fill the key, and compare group totals with the grand total.
  5. 5. Which is the best way to compare two computed float columns in a check?

    Floating-point results carry tiny errors. A tolerance-based comparison states how close is close enough.

Summary

  • Chained assignment (df[mask]["col"] = x) edits a temporary copy; do it in one .loc[mask, "col"] = x step.
  • With Copy-on-Write, subsets and column views are independent of their parent, so modify the frame itself.
  • Arithmetic and column assignment align on index labels; use fill_value, .to_numpy() or reset_index deliberately.
  • Mixed-type columns become object; convert with to_numeric(errors="coerce") and count the failures.
  • Missing values turn integers into floats; use nullable dtypes and isna(), never == np.nan.
  • groupby drops missing keys by default, and inplace=True is not a performance tool.
  • Compare floats with tolerances, round once, and store money as integer cents.

Next: Testing and Validating DataFrames