Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 57 ยท Estimation

Point Estimation

Learn point estimation: estimators, bias, variance, mean squared error, consistency, the method of moments and maximum likelihood, with simulations you can run in Python.

Estimation is the heart of statistics: using sample data to say something about an unknown population quantity. A point estimate is a single number offered as the best guess for a parameter: the sample mean for the population mean, the sample proportion for the population proportion, the sample standard deviation for sigma. In the next lessons we will add a margin of error around such a guess (a confidence interval). Here we ask a more basic question: what makes a point estimate good?

Estimators and estimates

An estimator is a rule (a formula) that turns a sample into a number. An estimate is the number you get from one particular sample. The estimator is a random variable (it changes with the sample); the estimate is just a number.

For example, x-bar = (x1 + ... + xn) / n is an estimator of mu. If your sample is 4, 7, 5, then 5.33 is the estimate.

Bias

An estimator T for a parameter theta has bias E[T] - theta. If the bias is 0 it is called unbiased: averaged over all possible samples, it hits the target.

  • The sample mean is unbiased for mu.
  • The sample proportion is unbiased for p.
  • The sample variance with divisor n - 1 is unbiased for sigma^2; the divisor n version is biased low.
  • The sample standard deviation s is slightly biased low for sigma (the square root bends averages).

Variance, standard error and MSE

An unbiased estimator can still be bad if it swings wildly. Precision is measured by the variance of the estimator; its square root is the standard error. For the mean, SE = sigma / sqrt(n).

To combine accuracy and precision into one number, use the mean squared error:

TEXT
MSE(T) = Var(T) + Bias(T)^2

A slightly biased estimator with small variance can have a lower MSE than an unbiased one with large variance. That trade-off is why shrinkage and regularization work in machine learning.

Python Comparing estimators of a normal mean

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All four are (nearly) unbiased for the mean of a normal population, but the sample mean has the smallest variance. The median is slightly less efficient, and using just one observation wastes the data.

Consistency

An estimator is consistent if it gets closer to the true value as n grows (formally, converges in probability to the parameter). The sample mean, by the law of large numbers, is consistent. Consistency is a minimum requirement: an estimator that does not improve with more data is rarely useful.

Python Consistency in action

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two ways to build estimators

Method of moments

Equate the sample moments to the theoretical ones and solve. A population mean mu is estimated by x-bar. For an exponential distribution with rate r, the mean is 1 / r, so r-hat = 1 / x-bar.

Maximum likelihood

The likelihood L(theta) is the probability (or density) of the observed data viewed as a function of the parameter. The maximum likelihood estimate (MLE) is the value of theta that makes the observed data most probable.

For n coin flips with k heads, L(p) = p^k (1 - p)^(n - k), which is maximized at p-hat = k / n: the sample proportion. For a normal sample, the MLE of mu is x-bar, and the MLE of sigma^2 divides by n (biased, but consistent).

Python Maximum likelihood on a grid

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In practice we maximize the log-likelihood because sums are easier to handle (and to compute) than products of many tiny probabilities.

Choosing an estimator

flowchart TD
    A["Candidate estimators"] --> B{"Unbiased?"}
    B -- "Yes" --> C["Prefer the smallest variance"]
    B -- "Slightly biased" --> D["Compare MSE"]
    C --> E["Consistent as n grows?"]
    D --> E

Bias, variance and consistency guide the choice.

A point estimate alone hides how uncertain it is. "The average is 52" is much less informative than "the average is 52, give or take 3". That is exactly what confidence intervals add.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. An estimator is unbiased if:

    Unbiased means E[T] = theta, so errors cancel out on average.
  2. 2. Which formula is correct for the mean squared error?

    MSE = Var(T) + Bias(T)^2.
  3. 3. A coin lands heads 14 times in 40 flips. The maximum likelihood estimate of p is:

    The MLE of a binomial p is the sample proportion 14/40 = 0.35.
  4. 4. A consistent estimator:

    Consistency means the estimator converges to the parameter as the sample size increases.

Summary

  • An estimator is a formula; an estimate is its value for one sample.
  • Bias is systematic error, variance is spread; MSE combines them.
  • Good estimators are consistent and, among unbiased ones, have the smallest variance.
  • Method of moments and maximum likelihood are the two standard ways to build estimators.

Next: Degrees of Freedom and the t Distribution