Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 76 ยท Statistics for Data Science

Descriptive Statistics in Practice

Choose the right summary: mean vs median, spread measures, percentiles, weighted averages and grouped summaries with pandas, and avoid common traps.

Descriptive statistics squeeze a column of thousands of values into a few numbers: where the values sit, how far apart they are, and what the extremes look like. They are the first thing you compute and the last thing a busy reader remembers, so choosing the right summary matters. The wrong one can tell a story that no individual in the data would recognize.

The pandas course already shows how to call describe, mean and friends. This lesson is about the judgement behind them: when the mean misleads, which measure of spread to report, how percentiles answer practical questions, why a weighted average is sometimes the only correct one, and how to summarize by group without fooling yourself. All examples use small generated tables with fixed seeds.

Center: mean, median and mode

The mean is the sum divided by the count. The median is the middle value after sorting; half the data are below it. The mode is the most frequent value (useful for categories and discrete counts).

The mean uses every value, which is its strength and its weakness: a few huge values pull it up. The median ignores how large the extremes are, so it is robust. Whenever data are skewed, as incomes, prices, response times and order values nearly always are, the two disagree, and the gap itself tells you about the skew.

Python Mean vs median on skewed data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

More than half of the people earn less than the mean, so "the average income" does not describe a typical person. Add one extreme value and the mean jumps while the median barely moves. Rule of thumb: use the median (and percentiles) for skewed data and the mean for roughly symmetric data or when totals matter (total revenue is the mean times the count, so the mean is the right basis for budgets even when it is not typical).

A skewness near 0 means symmetric; positive means a long right tail. If skewness is large, consider reporting the median, or analyzing a log-transformed version (see Feature Engineering).

Spread: how much do values vary?

A center alone is not a description. Two stores with an average of 100 sales per day can be steady or wildly erratic. The common measures of spread:

Measure What it is Robust?
Range max minus min No, depends on two values
Variance and standard deviation average squared (and its root) distance from the mean No
IQR 75th minus 25th percentile Yes
MAD median of absolute deviations from the median Yes
Coefficient of variation standard deviation divided by mean No

Use the standard deviation with the mean (for roughly bell-shaped data) and the IQR with the median. The coefficient of variation is useful for comparing variability of things measured on different scales, such as a price in dollars and a weight in kilograms, as long as values are positive and the mean is not near zero.

By default pandas' std uses ddof=1 (sample standard deviation), while NumPy's np.std uses ddof=0 (population). That is a classic source of small, confusing mismatches.

Python Same center, different spread

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two stores have almost the same mean and median, and entirely different standard deviation, IQR and range. Which store would you trust to meet a daily target? Only the spread tells you.

Percentiles answer practical questions

A percentile is the value below which a given share of the data falls: the 90th percentile is larger than 90% of observations. Percentiles turn abstract distributions into operational statements:

  • "95% of orders ship within 4 days" (a service-level promise)
  • "The top 10% of customers account for 60% of revenue" (concentration)
  • "Median page load is 1.2 s, but the 99th percentile is 9 s" (tail latency)

For things that can go wrong for a few users, report the tail, not the average. A site with a fast mean can still be unusable for 1 user in 100.

Python Percentiles for a service-level promise

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Note the difference between the mean (a single number that no order may actually take) and the 95th percentile (a concrete promise). The last line is a simple concentration measure; it is a common finding in revenue data.

Weighted statistics

A plain mean treats each row equally. Sometimes rows should count for different amounts, and a plain mean gives the wrong answer. The classic case is an average of averages: if you average the group averages, a group of 10 people counts as much as a group of 10,000.

A weighted mean is sum(weight * value) / sum(weight). Typical weights are group sizes, survey weights (to correct for over- and under-sampled groups), or importance (credit hours when computing a grade-point average). Weights belong to the definition of the quantity, not to optional polishing.

flowchart LR
    A["Group averages<br/>(per store)"] --> B["Plain mean of averages<br/>(wrong when sizes differ)"]
    A --> C["Weighted mean<br/>(weights = group sizes)"]
    D["All rows together"] --> C
    C --> E["Same as the overall mean"]

A weighted mean of group averages equals the overall mean; an unweighted mean of the averages does not.

Python Average of averages vs weighted average

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The plain mean is far too high because the small airport store, with its large baskets, counts as much as the busy downtown store. The weighted figure equals total revenue divided by total customers, which is what "average basket" actually means.

Grouped summaries

Real questions are almost always about groups: by region, plan, month. In pandas, groupby plus agg computes several summaries per group in one go, and named aggregation gives readable column names. Always include the count, because a mean over three rows and a mean over three thousand rows are not equally reliable.

Python groupby with named aggregation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The pivot table splits groups further, and the counts show where cells get thin. A summary for a cell with only a handful of rows is anecdote, not statistics.

Shape: skewness, kurtosis and fixing skew

Centre and spread leave one question open: is the distribution balanced around its centre, and how heavy are its tails? Choose the statistic by the kind of data first. Qualitative (categorical) data such as plan or city take counts and a mode; quantitative (numeric) data take everything in this lesson. Shape statistics only make sense for quantitative data.

Skewness measures asymmetry. A symmetric distribution has its mean close to its median. In a positively skewed one the long tail points right and the mean is pulled above the median (incomes, prices, response times). In a negatively skewed one the tail points left and the mean falls below the median (exam scores with a ceiling). The common formulas are:

  • Pearson's moment skewness, the average cubed standardized value; this is what pandas.Series.skew and scipy.stats.skew report.
  • Pearson's second coefficient, 3 * (mean - median) / std, a quick estimate from three familiar numbers.
  • Bowley's measure (quartile skewness), (Q3 + Q1 - 2 * median) / (Q3 - Q1), which uses only quartiles and so ignores outliers. It runs from -1 to +1.
  • Kelly's measure, the same idea with the 10th and 90th percentiles: (P90 + P10 - 2 * median) / (P90 - P10); it looks further into the tails.

Kurtosis measures tail weight and peak. A distribution is mesokurtic when it resembles a normal curve (excess kurtosis near 0), leptokurtic when the tails are heavier and the peak sharper (positive excess kurtosis, more outliers) and platykurtic when the tails are lighter and the shape flatter (negative, like a uniform distribution).

Python Four skewness measures and three kurtosis types

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All four skewness measures agree on the direction. Bowley's and Kelly's values are bounded and robust, which makes them good for comparing skew across variables that contain outliers.

Reducing skew

Many methods work better on roughly symmetric data. For positive skew, apply a power transform that squeezes large values more than small ones: a square root (mild, for counts), a cube root (stronger, and works with zero and negative values) or a logarithm (strongest, positive values only; use log1p if zeros occur). Box-Cox (scipy.stats.boxcox, positive data) and Yeo-Johnson (any values) choose the power for you through a parameter lambda. For negative skew, reflect first (max - x) and then transform. The alternative to transforming is to use a model that tolerates skew: generalized linear models (GLMs) with a gamma or Poisson family model skewed, positive responses directly, and tree-based models do not care about skew at all. Linear Regression and Modelling says when each is needed.

Python Transforms that reduce right skew

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Traps to avoid

  • Averaging percentages. The average of 10% (of 1,000) and 50% (of 10) is not 30%. Add up counts and divide, or use weights.
  • Mean of a ratio vs ratio of means. Average revenue-per-visit across days differs from total revenue divided by total visits. Decide which question you are asking.
  • Summaries that cannot be combined. Medians and standard deviations of groups cannot be combined into an overall median or standard deviation without the underlying data; sums and counts can.
  • Ignoring missing values. mean skips them silently. If 40% of values are missing, the result describes the other 60% only. See Handling Missing Values.
  • Reporting too many decimals. A mean of 42.37819 suggests precision that the data cannot support.
  • A summary without a picture. Four datasets can share the same mean, spread and correlation yet look completely different when plotted (the Anscombe lesson). Draw the data too; see Looking at One Variable at a Time.

Tip: A good minimum summary for any numeric column is: count, missing count, median, IQR (or mean and standard deviation for symmetric data), minimum and maximum. If you can add one chart, add a histogram.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A column of house prices has a few very expensive mansions. Which center is more typical?

    The mean is pulled toward extreme values in skewed data. The median stays with the middle of the data and describes a typical house better.
  2. 2. Which pair of center and spread goes together as robust measures?

    The median and the IQR depend on the middle of the data and are barely affected by outliers.
  3. 3. Store A has 1,000 customers with average spend 20; store B has 100 customers with average spend 50. What is the average spend per customer across both?

    Weighted by customers it is (100020 + 10050) / 1100 = about 22.7. The plain mean of 20 and 50 (35) ignores the group sizes.
  4. 4. Why include the count in a grouped summary?

    Statistics from small groups vary a lot. The count tells the reader how much to trust each row.
  5. 5. What is the difference between pandas std() and NumPy np.std() by default?

    The ddof defaults differ, so the results differ slightly, most visibly in small samples. Set ddof explicitly when it matters.

Summary

  • The mean is sensitive to outliers; the median is robust. For skewed data (money, durations) prefer the median and percentiles, but use the mean when totals matter.
  • Report a spread with every center: standard deviation with the mean, IQR or MAD with the median.
  • Percentiles turn distributions into practical statements such as service-level promises and tail behavior.
  • Use weighted means when rows or groups should count unequally; never average averages of groups of different sizes.
  • Group with groupby().agg(...), always include counts, and beware of tiny cells.
  • Pair summaries with a plot, and round to what the data can support.

Next: Probability and Distributions You Will Meet, the shapes behind these numbers.