Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 49 ยท Random Numbers and Statistics

Statistics, Correlation and Covariance

Compute mean, median, mode, standard deviation, covariance and correlation with NumPy, then normalize data, smooth it with a moving average and bin it in 2D.

Once data lives in an array, the first questions are always the same. What is a typical value? How spread out is it? Do two measurements move together? NumPy answers all of these with a handful of functions that work on whole arrays at once, so you can summarize a million numbers in one line.

This lesson covers the descriptive statistics you will use every day: center and spread, covariance and correlation, standardizing columns, smoothing a series with a moving average, and counting how two variables fall into a grid. If you need a refresher on reducing along axes, see Aggregations and Axes. The random data below comes from generators with fixed seeds, so every run prints the same numbers.

Center and spread

The mean and median are one method call or function away. NumPy has no mode function, but np.unique with return_counts=True gives you the building blocks.

Python Mean, median, mode and spread

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A few things to notice:

  • The mean is pulled by extreme values, the median is not. When they differ a lot, your data is skewed or has outliers.
  • The mode is the most frequent value. counts.argmax() returns the position of the first largest count, so ties resolve to the smallest value because np.unique returns values sorted.
  • np.average takes a weights array. Use it when some observations count more than others, such as a grade where exams outweigh quizzes.

Population versus sample standard deviation

std and var divide by N by default. That is right when your array is the entire population. When the array is a sample drawn from a larger population, dividing by N - 1 gives a less biased estimate. You request it with the ddof ("delta degrees of freedom") argument: the divisor is N - ddof.

NumPy defaults to ddof=0, while pandas' .std() defaults to ddof=1. The same numbers can give different answers in the two libraries, so always check which one you mean.

Covariance and correlation

Covariance measures whether two variables rise and fall together. A positive value means they tend to move in the same direction, a negative one means opposite directions. Its size depends on the units, which makes it hard to read. Correlation fixes that by dividing by both standard deviations, producing a number between -1 and 1.

flowchart LR
    A["Rows = variables<br/>columns = observations"] --> B["np.cov(data)"]
    A --> C["np.corrcoef(data)"]
    B --> D["Covariance matrix<br/>diagonal = variances"]
    C --> E["Correlation matrix<br/>diagonal = 1"]

Both functions treat each row as a variable and each column as an observation.

That layout is the opposite of the "rows are samples" convention used by most data tables. Pass rowvar=False if your variables are columns.

Python Covariance and correlation matrices

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the correlation matrix like a table: entry [i, j] relates variable i to variable j. The diagonal is 1 (a variable correlates perfectly with itself), and the matrix is symmetric. Here hours and score correlate strongly (about 0.91), while sleep is close to 0 for both because it was generated independently.

Correlation only captures linear relationships, and it never proves cause. Two variables can have a correlation near 0 and still be tightly related, for example y = x**2 on a range symmetric around 0.

Normalizing and z-scores

Columns with different units (centimeters and kilograms, say) are hard to compare or feed into algorithms. Two standard fixes are:

  • Z-score standardization: subtract the column mean and divide by the column standard deviation. The result has mean 0 and standard deviation 1.
  • Min-max scaling: map each column to the range 0 to 1.

Both rely on broadcasting: a (5, 2) array minus a (2,) array of column statistics just works.

Python Z-scores and min-max scaling

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A z-score tells you how many standard deviations a value sits from the mean. Values beyond about 3 in absolute value are often treated as outliers. Always compute mu and sd from your training data and reuse them on new data; recomputing on each new batch silently changes the scale.

Moving averages with convolve

Noisy series are easier to read after smoothing. A moving average replaces each point with the mean of a window around it. You can express it as a convolution with a kernel of equal weights [1/k, 1/k, ..., 1/k].

np.convolve(a, kernel, mode="valid") returns only the positions where the window fits fully, so the result is k - 1 elements shorter. The "same" mode keeps the length but blends in zeros at the edges, which distorts the first and last values.

Python A 3-point moving average

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The cumulative-sum version computes each window sum as the difference of two prefix sums, so its cost does not grow with the window size. For a window of 10,000 points on a long series, that is a large saving.

Two-dimensional histograms

np.histogram counts values in bins along one axis. np.histogram2d does the same on a grid, which shows how two variables are distributed together. It returns the counts plus the bin edges for each axis.

Python A 2D histogram of correlated data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Rows of counts correspond to x bins and columns to y bins, which is why the plot transposes the array (fine.T) before imshow, where rows are the vertical axis. The bright diagonal band is the positive correlation you built into the data.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does ddof=1 change in arr.std(ddof=1)?

    ddof is 'delta degrees of freedom'. The divisor is N - ddof, so ddof=1 gives the unbiased sample estimate; the default ddof=0 divides by N.
  2. 2. You have a table with one row per person and one column per measurement. How do you get the correlation between measurements?

    np.corrcoef treats rows as variables by default. With people in rows and measurements in columns you must pass rowvar=False (or transpose the table).
  3. 3. How can you find the mode of an integer array with NumPy?

    NumPy has no mode function. np.unique returns the sorted distinct values and their counts, and argmax of the counts selects the most frequent one.
  4. 4. What is the length of np.convolve(a, np.ones(5) / 5, mode='valid') when a has 100 elements?

    In 'valid' mode the output has len(a) - k + 1 elements, here 100 - 5 + 1 = 96.

Summary

  • mean, median, std, var and np.average(weights=...) summarize an array; compute the mode with np.unique(..., return_counts=True).
  • ddof=1 gives the sample standard deviation or variance; NumPy's default is ddof=0.
  • np.cov and np.corrcoef expect variables in rows; use rowvar=False for variables in columns. Correlation is unit-free and lies between -1 and 1.
  • Z-scores (x - mean) / std and min-max scaling are one-liners thanks to broadcasting.
  • np.convolve with an equal-weight kernel gives a moving average; mode="valid" avoids edge artifacts.
  • np.histogram2d counts pairs of values on a grid.

Next, put random numbers to work in Monte Carlo Simulations.