Skip to content
Tutorials Center
My learning

Topics

Lesson 41 of 49 ยท Performance and Internals

Common Pitfalls and Debugging Arrays

Debug NumPy like a pro: read shape errors, catch silent broadcasting bugs, integer overflow, view mutation, float equality, ambiguous truth values and NumPy 2 changes.

NumPy bugs come in two flavours. The friendly kind raises an exception with a message you can read: wrong shapes, bad axes, out-of-range indexes. The nasty kind produces a perfectly plausible answer that is simply wrong: a broadcast you did not intend, an integer that wrapped around, a change that leaked into another variable. The first kind costs minutes. The second can ship to production.

This lesson is a field guide to both. For each pitfall you will see a minimal example, the symptom, and a habit that prevents it. At the end you will meet np.testing, the assertion helpers built for arrays, and a checklist of what changed when NumPy moved from 1.x to 2.x, which matters whenever you read older tutorials or inherit an older codebase.

flowchart TD
    A["Wrong result or error"] --> B["Print shape and dtype of every input"]
    B --> C{"Shapes as expected?"}
    C -->|"no"| D["Fix reshape, axis or newaxis"]
    C -->|"yes"| E{"dtype as expected?"}
    E -->|"no"| F["Cast earlier or widen before math"]
    E -->|"yes"| G{"Did something change another array?"}
    G -->|"yes"| H["Views: copy explicitly"]
    G -->|"no"| I["Shrink to a tiny example and assert"]

A debugging order that finds most array bugs: shapes first, then dtypes, then aliasing, then a minimal example.

Reading shape errors

Most NumPy errors name the shapes involved. Learn to read them from left to right: first which operation failed, then which dimension, then the sizes. Run the examples below and compare the messages.

Python A catalog of common errors

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

What each message is telling you:

  • operands could not be broadcast together with shapes (3,4) (4,3): the two shapes disagree in a dimension where neither side is 1. Look at the trailing dimensions first. The rules are in broadcasting.
  • matmul: Input operand 1 has a mismatch in its core dimension 0: the inner sizes of a matrix product differ. For A @ B, the number of columns of A must equal the number of rows of B. See vectors and the @ operator.
  • cannot reshape array of size 12 into shape (5,3): the element counts differ (12 versus 15). A -1 can fill in one unknown, but only if the total divides evenly.
  • all the input array dimensions except for the concatenation axis must match: you are joining along one axis, so every other axis must be equal. Check which axis you really want in joining and splitting.
  • index 3 is out of bounds for axis 0 with size 3: valid indexes are 0 to 2. Note the message names the axis.
  • axis 2 is out of bounds for array of dimension 2: a 2D array has only axes 0 and 1.

The single most useful debugging habit is to print x.shape and x.dtype for every variable involved, right before the failing line. Add temporary assert statements that state the shape you believe you have: assert X.shape == (n, 3).

Silent broadcasting bugs

Broadcasting never raises an error when shapes are compatible, even when the combination is not what you meant. The classic accident mixes a 1D array, a column vector (n, 1) and a row, and produces an (n, n) matrix instead of an (n,) vector.

Python A column vector turns a product into a matrix

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both wrong results are valid floating point numbers, so nothing warns you. Defences: assert shapes at function boundaries, check the shape of the result of the first operation in a new formula, and prefer explicit reshape or np.newaxis over relying on luck. A quick test is to run the code on a tiny input where you can compute the answer by hand.

Integer overflow

NumPy integers have a fixed size. When a result does not fit, NumPy wraps around silently for array arithmetic (it does not raise an error and gives no warning).

Python Wraparound in uint8 and int32

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Image code is the usual victim: adding brightness to a uint8 image makes bright pixels turn dark. Cure it by widening the dtype before the arithmetic and clipping afterwards. Only creating a scalar or Python-int operand that cannot fit (as with np.uint8(300)) raises OverflowError. Be careful with an explicit dtype= on a reduction such as np.sum(x, dtype=np.uint8), which forces the small accumulator back. See data types.

Views that change your data

Slices are views, so writing through one writes into the original. Fancy indexing returns copies, but assigning through a fancy index does modify the original. Mixing the two up leads to bugs in both directions.

Python Views, copies and surprising mutation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This is the number-one source of "my input array changed" bugs. If a function must not alter its argument, either work on x = np.array(arg) (which copies) or use out-of-place operations (y = x * 10, not x *= 10). Views and copies explains which operations return which.

Float equality and ambiguous truth values

Floating-point numbers are binary approximations, so == on computed floats is unreliable. Use np.isclose and np.allclose with sensible tolerances, as covered in comparing arrays and floating point. Two other traps live in if statements: an array in a boolean context, and chained comparisons.

Python Float comparison and the ambiguous truth value

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

  • if a > 1: fails because an array of several booleans is neither true nor false. Say what you mean: .any(), .all(), or .size.
  • and, or and not need single truth values. For arrays use &, | and ~, with parentheses because those operators bind tighter than comparisons.
  • NaN is not equal to anything, not even itself. Use np.isnan.

Testing arrays with np.testing

The plain assert a == b does not work for arrays. numpy.testing offers assertions that compare shapes, values and tolerances, and print a helpful report of the mismatch.

Python np.testing assertions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Passing assertions print nothing; failures raise AssertionError with counts of mismatched elements and the first differences. In a Pytest suite, write np.testing.assert_allclose(result, expected, rtol=1e-7) and you get the same report in the test output.

Upgrading from NumPy 1.x to 2.x

NumPy 2 is the version this tutorial teaches. If you read older code, expect these differences:

Old (1.x) NumPy 2
np.float_, np.complex_, np.NaN, np.Inf, np.PINF np.float64, np.complex128, np.nan, np.inf
np.product, np.cumproduct, np.round_, np.asfarray np.prod, np.cumprod, np.round, np.asarray(x, dtype=float)
np.in1d, np.trapz, np.row_stack np.isin, np.trapezoid, np.vstack
np.random.seed, np.random.rand rng = np.random.default_rng(seed)
repr prints 3.0 for a float64 scalar repr prints np.float64(3.0)
value-based casting of scalars NEP 50: Python numbers adopt the array's dtype
np.array(x, copy=False) means "copy only if needed" copy=None means that; copy=False raises if a copy is required

Some of these names were deprecated in 2.0 and removed in a later 2.x release, so what you see depends on the exact version. The run below checks the current install.

Python What NumPy 2 removed and changed

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The biggest behavioural change is NEP 50. In NumPy 1.x, np.int8 array plus the Python number 1000 quietly picked a bigger type by looking at the value. In NumPy 2, a Python int or float is "weak": it takes the dtype of the array it meets, and raises OverflowError if it cannot fit. That makes results depend on dtypes only, never on values, but it also means that code relying on value-based upcasting must now cast explicitly. More on promotion in converting and casting types.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which statement about integer overflow in NumPy array arithmetic is true?

    Arithmetic on integer arrays wraps around modulo 2 to the power of the bit width, with no error or warning. Widen the dtype or clip if you need other behaviour.
  2. 2. scores has shape (3,) and w has shape (3, 1). What is the shape of scores * w?

    Broadcasting stretches (3,) to a row and (3, 1) to a column, producing a 3 by 3 table. It does not raise an error, which is why it hides bugs.
  3. 3. Why does if arr > 0: raise a ValueError for a multi-element array?

    NumPy refuses to guess whether you mean any or all, so you must say which.
  4. 4. sub = a[2:5]; sub[:] = 0. What happens to a?

    Basic slices share memory with the original array. Use .copy() when you need an independent array.
  5. 5. Which is the right way to check that two float arrays match within rounding error in a test?

    assert_allclose uses a tolerance and prints a helpful report of the mismatched elements when it fails.

Summary

  • Read shape errors by naming the operation, the dimension and the sizes; print shape and dtype of all inputs before guessing.
  • Broadcasting can silently produce the wrong shape (for example (3,) * (3, 1) gives (3, 3)); assert shapes at boundaries.
  • Fixed-size integers wrap around on overflow; widen the dtype first and clip when needed.
  • Basic slices are views; fancy indexing copies, but assignment through it modifies the original. Use .copy() to protect data.
  • Compare floats with np.isclose and np.allclose; use .any(), .all(), &, | and ~ with arrays.
  • Use np.testing.assert_allclose and assert_array_equal in tests.
  • NumPy 2 removed many aliases (np.float_, np.NaN, np.product, ...), changed scalar repr, and adopted NEP 50 promotion.

Next: Project: Conway's Game of Life