Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 38 ยท Statistics

Distributions and Descriptive Statistics

Use scipy.stats distribution objects: pdf, cdf, ppf, rvs, loc and scale, frozen distributions, describe and summary statistics, and fit a distribution to data.

Statistics starts with two questions: what does my data look like? and what process could have produced it? The first is answered by summaries (the mean, the spread, the shape). The second needs a probability distribution: a mathematical description of how likely each outcome is, such as the bell curve for heights or the Poisson distribution for the number of emails per hour.

scipy.stats contains more than a hundred distributions, all with the same interface. Learn it once and you can work with the normal, binomial, exponential, gamma, t, chi-square and every other distribution in the same way. This lesson covers that interface, the descriptive functions for samples, and fitting a distribution to data. The next lessons build on it: Hypothesis Testing and Correlation, Regression and Bootstrap.

flowchart LR
    P["parameters<br/>loc, scale, shape"] --> D["distribution object"]
    D -->|"pdf / pmf"| A["density at x"]
    D -->|"cdf, sf"| B["P(X โ‰ค x), P(X > x)"]
    D -->|"ppf, isf"| C["x for a given probability"]
    D -->|"rvs"| E["random samples"]

One object, five kinds of question. The ppf is the inverse of the cdf.

The distribution interface

Every distribution in scipy.stats is an object with the same methods:

Method Answers
pdf(x) (continuous) or pmf(k) (discrete) How dense/likely is the value x?
cdf(x) What is the probability of a result at most x?
sf(x) The "survival function": probability of a result greater than x (1 - cdf, but more accurate in the far tail)
ppf(q) The value below which a share q of the probability lies (the quantile, inverse of cdf)
isf(q) Inverse of sf
rvs(size=n) n random draws
mean(), var(), std(), median(), interval(c) Moments and the central interval holding probability c

For a continuous distribution the density pdf is not a probability (it can exceed 1); probabilities are areas under it, which is what cdf computes.

Python The standard normal distribution

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last two lines show why sf exists: for extreme values 1 - cdf rounds to zero, while sf keeps the true tiny probability. It also matters for p-values later.

loc, scale and frozen distributions

Distributions come in a standard form, and you adjust them with two universal arguments: loc shifts the distribution and scale stretches it. For the normal, loc is the mean and scale is the standard deviation. Other distributions may need extra shape parameters first (stats.gamma(a=2.5), stats.t(df=5)).

You can pass all parameters on every call, but it is cleaner to freeze the distribution once: call the distribution with its parameters, get back an object with the parameters remembered, and use its methods without repeating them.

Python Frozen distributions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The t distribution with few degrees of freedom has heavier tails than the normal, so extreme values are far more likely: the last line shows an extreme outcome more than twenty times as probable. That detail matters when you test hypotheses on small samples.

Discrete distributions

Discrete distributions (binomial, Poisson, geometric, ...) count things. They use pmf(k) for the probability of exactly k and cdf(k) for at most k. Here ppf returns the smallest integer whose cdf reaches the requested probability.

Python Binomial and Poisson

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Discrete distributions use pmf (probability mass), continuous ones use pdf (probability density). Calling pdf on a binomial raises an error, which is a useful reminder of the difference.

Random samples and reproducibility

rvs(size=n) draws random samples. To make the numbers repeatable, pass random_state. It accepts an integer seed or, preferably, a NumPy generator, which you can then share across calls:

Python Drawing samples

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The sample mean and standard deviation are close to the true values but not equal: they are estimates from finite data, which is the starting point of all statistical inference.

Describing a sample

Given actual data you reach for summary statistics. stats.describe bundles the common ones; individual functions give more control:

  • stats.describe(x): count, min and max, mean, variance (with ddof=1 by default, the sample variance), skewness and excess kurtosis.
  • stats.skew(x): asymmetry. Positive means a long right tail.
  • stats.kurtosis(x): tail heaviness relative to a normal distribution (0 for a normal one, positive for heavy tails).
  • stats.iqr(x): the interquartile range, a spread measure resistant to outliers.
  • stats.sem(x): the standard error of the mean, std / sqrt(n).
  • stats.zscore(x): how many standard deviations each value is from the mean.
  • stats.trim_mean(x, 0.1): the mean after dropping the lowest and highest 10%.
Python Summaries and the effect of an outlier

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A single bad measurement drags the mean by about two units, while the median and the trimmed mean barely notice. When your data may contain outliers, report a robust statistic alongside the mean.

Percentiles, ranks and circular data

Four more summary tools appear constantly in practice.

Percentiles of a sample. stats.scoreatpercentile(x, 90) returns the value below which 90 percent of the sample lies (the same idea as np.percentile, with an interpolation_method choice and the ability to restrict to a limit range). Its inverse is stats.percentileofscore(x, score), which answers "what share of my sample is at or below this score?" The kind argument decides how ties count: "rank" (average, the default), "weak" (at or below), "strict" (strictly below) and "mean" (average of weak and strict).

Ranks. stats.rankdata(x) replaces every value by its rank, the basis of rank-based tests and of Spearman correlation. Ties get the average rank by default; method="min", "max", "dense" and "ordinal" handle ties differently.

Circular statistics. Angles and clock times do not behave like ordinary numbers: the average of 350 degrees and 10 degrees is 0 degrees (north), not 180. stats.circmean(samples, high=360, low=0), circvar and circstd compute the mean and spread of values on a circle. The high and low arguments set the period: high=24 for hours, high=2*np.pi (the default) for radians.

NaN handling and keepdims. Most functions in scipy.stats take nan_policy: "propagate" (the default, the result is NaN if any input is), "omit" (ignore NaN values) or "raise" (stop with an error). Many also take axis= and keepdims=True, which keeps the reduced axis with length 1 so the result broadcasts back against the original array.

Python Percentiles, ranks, circular means and nan_policy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The ordinary mean of the four bearings is 180, exactly opposite the truth, while the circular mean lands at about 0 degrees. The same trap hides in wind direction, time of day, day of the year and phase angles, so use the circular functions whenever your numbers wrap around.

Comparing a sample with a distribution: Kolmogorov-Smirnov

The Kolmogorov-Smirnov (KS) test compares two cumulative distributions and reports the largest vertical gap D between them. It needs no assumption about the shape of the data. stats.kstest has two uses:

  • One sample: stats.kstest(sample, stats.norm(mu, sigma).cdf) tests whether the sample comes from a given distribution. The second argument is a distribution name or, as here, a CDF function (a frozen distribution's .cdf works well). The null hypothesis is "yes".
  • Two samples: stats.ks_2samp(a, b) tests whether two samples come from the same distribution, without saying which.

A small p-value is evidence against the null hypothesis. One caution: if you estimate the distribution's parameters from the same sample and then test against it, the p-value is too optimistic, because the fitted curve was tailored to the data. For a normality check with estimated parameters prefer stats.shapiro or stats.goodness_of_fit.

Python One-sample and two-sample KS tests

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Samples from the right distribution give large p-values (no evidence of a difference); the exponential sample is rejected as normal with a p-value close to zero. The statistic field of the result is D itself, so it also works as an effect size: 0 means identical CDFs, 1 means no overlap at all. The skewness test stats.skewtest and kurtosis test stats.kurtosistest check the two moments separately, and stats.jarque_bera combines them, which is handy for large samples such as financial returns.

Fitting a distribution to data

Every continuous distribution has a .fit(data) method that estimates its parameters by maximum likelihood. It returns a tuple of shape parameters (if any), then loc, then scale. Two tips:

  • Fix parameters you know with floc=0 (location) or fscale=1. Positive-only distributions such as gamma, lognormal and exponential should almost always use floc=0; otherwise the fit may wander to a strange shifted solution.
  • To compare candidate distributions, compute the log-likelihood np.sum(dist.logpdf(data, *params)) and the AIC, 2 * k - 2 * loglik, where k is the number of free parameters. Lower AIC is better.
Python Which distribution describes this data?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The gamma fit has the lowest AIC and recovers parameters reasonably close to the truth; the normal fit is clearly worse because the data is skewed and cannot go below zero. AIC only ranks the candidates you offered, so always look at the plot as well, and remember that a good fit is evidence, not proof, of the true process.

Warning: A fit with the default arguments may be unstable on positive-only data. Use floc=0 whenever zero is a natural lower bound, and never trust a fitted shape parameter from a handful of points.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does stats.norm.ppf(0.975) return?

    ppf is the percent point function, the inverse of cdf. For the standard normal it returns about 1.96.
  2. 2. Why use sf(x) instead of 1 - cdf(x) for extreme values?

    The two are mathematically equal, but when cdf is extremely close to 1 the subtraction rounds to zero, while sf computes the small tail directly.
  3. 3. How do you describe a normal distribution with mean 100 and standard deviation 15?

    All SciPy distributions use loc and scale; for the normal, loc is the mean and scale is the standard deviation.
  4. 4. A binomial distribution object has no pdf. Which method gives P(X = 3)?

    Discrete distributions have a probability mass function pmf(k) for exactly k; cdf(3) would give P(X โ‰ค 3).
  5. 5. You fit a gamma distribution to positive data and the result has a strange location far from zero. What should you do?

    Positive-only distributions should usually have their location fixed at zero. Otherwise the optimizer can trade location against shape and give an unstable fit.

Summary

  • All scipy.stats distributions share one interface: pdf/pmf, cdf, sf, ppf, isf, rvs and moment methods.
  • loc shifts and scale stretches; shape parameters (a, df, ...) come first. Freeze a distribution with stats.norm(loc=..., scale=...) to avoid repeating parameters.
  • Use sf for tail probabilities and ppf to go from a probability to a value.
  • rvs(size=n, random_state=...) draws reproducible samples.
  • describe, skew, kurtosis, iqr, sem, zscore and trim_mean summarize a sample; robust statistics resist outliers.
  • .fit(data) estimates parameters by maximum likelihood; fix floc=0 for positive data and compare candidates by AIC and by plotting.

Next up: decide whether an observed difference is real in Hypothesis Testing.