Skip to content
Tutorials Center
My learning

Topics

Lesson 13 of 44 ยท Measurement and Sampling

Reliability and Validity of Measures

How to judge a psychological measure: test-retest, internal consistency and inter-rater reliability, plus content, criterion and construct validity, with simulations of Cronbach's alpha.

When you step on a bathroom scale, you want two things. The reading should not change if you step off and on again (reliability), and it should match your true weight (validity). Psychological measures have the same two requirements, but there is no bathroom scale for intelligence or anxiety. We have to build instruments and then show that they are good.

This lesson covers how to check reliability with several methods, how to think about validity, and how the two relate. You will compute Cronbach's alpha in a small simulation.

Four dartboards showing reliable and valid, reliable but not valid, valid but not reliable, and neither.
The dartboard picture: reliability is a tight cluster, validity is the cluster landing on the bullseye.

What you will learn

  • What reliability means and three ways to estimate it.
  • What validity means, including content, criterion and construct validity.
  • Why reliability is necessary but not sufficient for validity.
  • How to read a reliability coefficient, and how Cronbach's alpha is computed.

Key terms

Reliability: the consistency of a measure.

Test-retest reliability: the correlation between scores from the same people at two times.

Internal consistency: how well the items in a scale agree with each other.

Inter-rater reliability: agreement between different observers or scorers.

Validity: the extent to which a measure captures the concept it is meant to capture.

Reliability

Every measurement is the true value plus error. Reliability is the share of the differences between people that comes from real differences, not from error. It is usually reported as a correlation coefficient between 0 and 1, where higher is more consistent.

Test-retest

Give the same test to the same people twice, a week or two apart, and correlate the scores. It suits stable traits such as personality or intelligence. It is a poor fit for fleeting states such as mood. If the gap is too short, people may remember their answers, and if it is too long, the trait may really change.

Internal consistency

A questionnaire has several items meant to measure the same thing. If they do, people who score high on one item should tend to score high on the others. Cronbach's alpha (Lee Cronbach, 1951) summarizes this:

alpha = k / (k - 1) x (1 - sum of item variances / variance of total scores)

where k is the number of items. Values of around 0.7 or higher are commonly taken as acceptable for research, and higher for decisions about individuals. Alpha rises with the number of items, so a very high alpha on a long scale can simply mean redundancy.

Inter-rater reliability

When people judge behavior, such as coding an essay or an observed interaction, different raters must agree. For categories, researchers use Cohen's kappa (Jacob Cohen, 1960), which corrects raw percent agreement for the agreement expected by chance. This returns in the lesson on observation.

Validity

A measure can be consistent and still wrong. Validity asks whether it measures the target concept. Several kinds of evidence help.

Type Question Example
Face validity Does it look like it measures the concept? A depression questionnaire asks about mood
Content validity Does it cover all aspects of the concept? An exam covering only one chapter lacks content validity
Criterion validity Does it predict or match a relevant outside standard? An aptitude test predicting later job performance
Construct validity Does it behave as theory says it should? A new anxiety scale correlates with other anxiety measures but not with shoe size

Construct validity is the umbrella, a concept developed by Lee Cronbach and Paul Meehl in 1955. It is built up from many pieces of evidence, including two especially useful ones.

  • Convergent validity: scores relate strongly to other measures of the same or similar constructs.
  • Discriminant validity: scores relate weakly to measures of unrelated constructs.

Validation is never finished. It is an ongoing accumulation of evidence, and it is specific to a purpose and a population. A measure validated in one country may not work the same way in another.

How reliability and validity relate

Reliability is necessary for validity but does not guarantee it. A measure full of random error cannot correlate well with anything, so it cannot be valid. But a very consistent measure can be consistently measuring the wrong thing. Measuring head circumference is highly reliable, and as a measure of intelligence it is invalid.

Improving reliability

  • Use more items, written clearly and unambiguously.
  • Standardize instructions and conditions.
  • Train raters and check agreement.
  • Pilot-test and drop poor items.
  • Measure under similar conditions each time.

Common misconceptions

"High reliability means high validity." Reliable measures can still measure the wrong thing.

"Alpha of 0.9 is always better than 0.8." Very high alpha can mean redundant items, and what counts as enough depends on how the scores will be used.

"Validity is a yes or no property of a test." It depends on the purpose, the group and the accumulated evidence.

Try it

Python Cronbach's alpha and test-retest (simulation)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Python Convergent and discriminant evidence (simulation)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A test gives nearly the same score for each person on two occasions two weeks apart. This shows good

    Consistency across time for the same people is test-retest reliability.
  2. 2. What does Cronbach's alpha estimate?

    Alpha summarizes internal consistency among items.
  3. 3. Measuring head circumference to assess intelligence would be

    Head circumference is measured consistently but does not capture intelligence.
  4. 4. A new anxiety scale correlates strongly with an established anxiety scale. This is evidence of

    Convergent validity means strong relations with measures of the same or similar construct.
  5. 5. Why is reliability necessary but not sufficient for validity?

    A noisy measure cannot be valid, but a consistent measure can still target the wrong concept.

Summary

  • Reliability is consistency, checked by test-retest, internal consistency (Cronbach's alpha) and inter-rater agreement.
  • Validity is whether a measure captures the intended concept, supported by content, criterion and construct evidence.
  • Convergent and discriminant patterns are key evidence for construct validity.
  • Reliability is necessary but not sufficient for validity.

Next: Sampling and Populations