Skip to content
Tutorials Center
My learning

Topics

Lesson 41 of 64 ยท Statistics and Analysis

Descriptive Statistics

Summarize data with pandas: mean, median, std, quantile, mode, skew, describe(), value_counts, nunique, corr, cov and weighted averages, with live examples.

Before you model, plot or report anything, you need to know what your numbers look like. How large are they typically? How much do they vary? Are a few extreme values dragging the average? Descriptive statistics answer those questions with a handful of numbers, and pandas computes them with one method call each.

Almost every method in this lesson works on a Series (one column) and on a DataFrame (one result per numeric column). All of them skip missing values by default, which is usually what you want; pass skipna=False if a single NaN should poison the result. If the data types of your columns are still unclear, take a look at Data Types and Conversion first.

Centre and spread

The centre tells you what is typical; the spread tells you how far values wander from it. The example builds 200 synthetic employees with a fixed random seed, then deliberately adds three extreme salaries so you can see which statistics notice them.

Python Mean, median and friends

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the output carefully. The mean (58,234) is pulled above the median (56,100) by the three huge salaries, and the standard deviation is a wild 25,371 although most people earn within a few thousand of the median. The IQR (interquartile range, the distance between the 25th and 75th percentiles) is only 11,950 because it ignores the extremes. When a distribution has outliers or a long tail, the median and the IQR describe it better than the mean and the standard deviation.

Here is what each method means:

Method Answers Notes
mean() the arithmetic average sensitive to outliers
median() the middle value (50th percentile) robust
mode() the most frequent value(s) always returns a Series, because there can be ties
std(), var() typical distance from the mean sample statistics (ddof=1), unlike NumPy's default
min(), max(), sum(), prod() extremes and totals
quantile(q) the value below which a fraction q of the data lies q can be a list
skew() asymmetry: positive means a long right tail 0 is symmetric
kurt() how heavy the tails are compared to a normal curve 0 is normal-like (excess kurtosis)
idxmax(), idxmin() the index label of the max / min handy to find which row

pandas' std() divides by n - 1 (a sample standard deviation), while np.std() divides by n by default. For the same data they differ slightly. Pass ddof=0 to pandas or ddof=1 to NumPy when you need to match.

The one-call summary: describe()

describe() bundles count, mean, standard deviation, minimum, quartiles and maximum for numeric columns. Add include="all" to also summarize text columns (number of unique values, the top value and its frequency), or choose your own percentiles:

Python describe() options

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

describe() returns an ordinary DataFrame, so you can transpose it with .T, select a row with .loc["50%"] or export it like any other table. Combined with groupby, as in the last call, it gives you a full statistical profile per group in one line. For choosing your own set of statistics, agg(["mean", "median", "std"]) is more flexible.

Counting categories

Numbers describe quantities; for categories you count. value_counts() is probably the most used method in all of pandas:

Python value_counts and nunique

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Use normalize=True for percentages, sort=False to keep the order of first appearance, and dropna=False when counting missing values is part of the question. nunique() counts distinct values and is a fast way to spot identifier columns (one distinct value per row) and constants (just one value). unique() returns the distinct values themselves.

Relationships between columns

corr() computes the pairwise correlation: a number between -1 and 1 that says how closely two columns move together in a straight line. cov() returns the covariance, the same idea without rescaling, so it depends on the units of your columns.

Python Correlation and covariance

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The micropip.install("scipy") line is only needed in the browser: method="spearman" uses SciPy, which pandas does not install for you (on your own computer run pip install scipy). Hours studied and the score are almost perfectly correlated (0.997). The Spearman option ranks the values first and so measures any consistently rising relationship, not only straight lines; here it is exactly 1.0 because the score rises with every extra hour. Always remember the classic warnings: correlation is not causation, it only sees linear patterns by default, and a single outlier can create or destroy it. Look at a scatter plot (see Plotting with pandas) before you trust a coefficient. corr() only uses numeric columns; on a DataFrame with text columns, select the numeric ones first with df.select_dtypes("number").

Weighted averages

mean() gives each row the same importance. Often rows should count differently, for example a price averaged over the number of units sold at it. pandas has no weighted_mean method, but the formula is short: multiply each value by its weight, sum, and divide by the sum of the weights.

Python Weighted mean, overall and per group

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The plain mean says the typical price is 20, but nearly all units sold at 10, so the weighted mean is 11.081, a much more honest "average price paid". The group-wise recipe (sum first, divide once) avoids the common mistake of averaging group averages, which is wrong when groups have different sizes. See agg, transform and filter for more ways to compute per-group results.

An average of averages is not the overall average. If store A sold 1,000 items at an average of 5 and store B sold 10 items at an average of 50, the mean of the two averages is 27.5, while the real average price per item is about 5.45. Weight by the counts, or compute from the raw rows.

More statistics: standard error, absolute deviation, mode and shape

A few more helpers complete the toolbox.

var() is the variance: the average squared distance from the mean, in squared units (dollars squared for salaries). By default pandas divides by n - 1 (the sample variance, ddof=1); pass ddof=0 to divide by n. std() is simply its square root, so std(ddof=0) goes with var(ddof=0). var works on a Series, on every numeric column of a DataFrame, and across rows with axis=1; agg(["var", "std", "sem"]) gives all three side by side. For the series [4, 8, 6, 5, 3, 9] the sample variance is about 5.37, the population variance about 4.47.

sem() is the standard error of the mean: std() / sqrt(n). The standard deviation describes how much single values vary; the standard error describes how precisely the mean is known, and it shrinks as the sample grows. Multiply it by about 1.96 to get a rough 95% margin around the mean.

Mean absolute deviation. Older pandas had Series.mad(), but it was removed in pandas 2.0. Calculate it by hand: the average distance from the mean, (s - s.mean()).abs().mean(). It is less sensitive to outliers than the standard deviation, because distances are not squared. For an even more robust measure, use the median absolute deviation: (s - s.median()).abs().median().

mode() returns every value that ties for most frequent, sorted. On a DataFrame it returns one column per original column, padded with NaN when columns have different numbers of modes. dropna=False counts missing values as a possible mode.

skew() and kurt() summarise shape. Skew above 0 means a long right tail (a few big values), below 0 a long left tail. Kurtosis above 0 means heavier tails than a normal curve, so more extreme values than you would expect.

value_counts(bins=...) counts numeric values by range, like a quick histogram table, without creating a binned column first.

Python sem, absolute deviation, mode, shape and binned counts

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The standard deviation is about the same for both samples (it estimates the real spread of 10), but the standard error is about nine times smaller in the larger sample. It has 100 times more values, and the error shrinks with the square root of the sample size, so 100 times the data gives a mean that is about ten times more precise. The outlier 40 inflates the standard deviation of data, while the median absolute deviation barely notices it. The exponential sample has a clearly positive skew, while the normal one sits close to 0 for both numbers.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A salary column contains a few extremely large values. Which pair of statistics describes the typical employee better?

    The median and the IQR depend on the ordering of values, not their size, so a few extremes barely move them. The mean and standard deviation are pulled towards the outliers.
  2. 2. What does df['city'].value_counts(normalize=True) return?

    normalize=True turns counts into proportions. By default the result is sorted by size in descending order.
  3. 3. Why does Series.mode() return a Series rather than a single number?

    If two values appear equally often, both are modes. mode().iloc[0] picks the first one when you need a single value.
  4. 4. You have group averages and group sizes. How do you compute the correct overall average?

    That is a weighted mean. Averaging averages treats a group of 10 rows as important as a group of 1,000.
  5. 5. Which statement about corr() is true?

    The default Pearson coefficient captures linear association only. Curved relationships can have a correlation near 0, and correlation never proves causation.

Summary

  • Centre: mean(), median(), mode(). Spread: std(), var(), quantile(), max() - min(), and the IQR. Shape: skew() and kurt().
  • Outliers move the mean and standard deviation a lot, the median and IQR hardly at all.
  • describe() gives a quick numeric profile (include="all" adds text columns); combine it with groupby for per-group profiles.
  • value_counts() (with normalize, dropna and bins), nunique() and unique() summarize categories.
  • sem() is the standard error of the mean; compute mean or median absolute deviation by hand because mad() was removed; mode() returns all ties.
  • corr() and cov() measure linear relationships; check a plot before trusting them.
  • Weighted averages need sum(value * weight) / sum(weight); never average averages.

Next: Exploratory Data Analysis Workflow strings these tools into a repeatable first look at any new dataset.