Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 47 ยท Defining and Classifying Disorders

Reliability and Validity of Diagnosis

Understand reliability and validity in psychiatric diagnosis, Rosenhan's pseudopatient study and its critics, kappa agreement, comorbidity, labelling and stigma.

If two doctors examine the same patient and reach different diagnoses, something is wrong with the system, not necessarily with the doctors. And if they agree perfectly but the category they agree on does not correspond to anything real, something is wrong in a deeper way. These two worries are the problems of reliability and validity, and they have shaped every revision of the diagnostic manuals since 1980.

Four dartboard-style targets: tight on the bullseye, tight off center, scattered around center, scattered off center
Caption: Reliability is consistency; validity is hitting the real target. A diagnosis can be reliable and still not valid.

What you will learn

  • The meaning of reliability and validity, and the main types of each.
  • How agreement is measured with Cohen's kappa.
  • What Rosenhan's 1973 study claimed, and why its critics matter.
  • Comorbidity, labelling and stigma.

Key terms

Reliability: consistency of a diagnosis, for example agreement between different clinicians (inter-rater) or over time (test-retest).

Validity: whether a diagnosis measures a real, meaningful condition: it predicts course, treatment response or underlying causes.

Cohen's kappa: agreement between raters corrected for the agreement expected by chance; 0 means chance, 1 means perfect.

Comorbidity: having two or more diagnoses at the same time.

Labelling: how a diagnosis changes how a person is seen and treated.

Reliability

Inter-rater reliability asks: do different clinicians give the same diagnosis? Before DSM-III, the answer was often no. A 1970s study of US and UK psychiatrists watching films of patients found that Americans diagnosed schizophrenia far more often than the British, who diagnosed more mania and depression. Part of the motive for explicit criteria in 1980 was to fix this.

Agreement is measured with Cohen's kappa. Percent agreement can mislead, since two raters will agree often just by chance if a disorder is rare. Kappa subtracts the chance agreement. As a rough guide, values below 0.4 are poor to fair, 0.4 to 0.6 moderate, 0.6 to 0.8 good, and above 0.8 very good. In the DSM-5 field trials (reported by Regier and colleagues in 2013), some diagnoses such as major depressive disorder and generalized anxiety disorder showed low kappa values (around 0.2 to 0.3, reported), whereas others such as autism spectrum disorder and PTSD showed better agreement. The trials sparked debate about how much agreement can be expected in real clinics, but they made clear that reliability is a continuing problem.

Test-retest reliability asks whether the same clinician gives the same diagnosis on two occasions. Both depend on how the assessment is done: structured interviews improve agreement over unstructured ones.

Validity

A diagnosis is valid to the extent that it identifies something meaningful. Several kinds are examined:

  • Descriptive validity: do the symptoms cluster together?
  • Predictive validity: does the label predict course and outcome?
  • Construct validity: does it relate in the expected ways to other measures and causes?
  • Etiological validity: do cases share a cause?
  • Concurrent validity: does it agree with other well-established measures?

The psychiatrists Eli Robins and Samuel Guze (1970) proposed that a valid diagnosis should be supported by clinical description, laboratory study, delimitation from other disorders, follow-up studies and family studies. Few psychiatric diagnoses meet all of these. Many categories have fuzzy boundaries and show substantial overlap with neighbors, which is why researchers talk about dimensions.

Rosenhan's pseudopatients

In 1973 David Rosenhan published "On Being Sane in Insane Places" in Science. Eight healthy people (the "pseudopatients") presented at 12 US hospitals complaining of hearing voices saying words like "empty", "hollow" and "thud". Once admitted, they behaved normally and said they felt fine. All were admitted, all but one with a diagnosis of schizophrenia, and stayed from about 7 to 52 days (an average of 19). They were discharged with "schizophrenia in remission". Staff rarely noticed that they were sane, though other patients sometimes did. In a second part, a hospital was told to expect pseudopatients; staff then labelled many real new patients as probable impostors, though none had been sent.

The study seemed to show that psychiatric diagnosis was unreliable and that labels color everything. It had enormous influence on opinion and policy. But its critics are important. Robert Spitzer (1975) argued that the results actually showed good judgment: the pseudopatients lied about a real symptom, and doctors responded sensibly, labelling them "in remission" at discharge. Hospitals would not be expected to detect deceit. More recently, journalist Susannah Cahalan (The Great Pretender, 2019) examined Rosenhan's papers and questioned whether the data and details were as reported. The practical conclusion is moderate: the study is a vivid demonstration of how labels and settings shape perception, but it is weak evidence about modern diagnostic reliability, and its own reliability is in doubt.

Comorbidity and heterogeneity

If categories were clean, people would usually have one diagnosis. Instead comorbidity is common: someone with depression often also meets criteria for an anxiety disorder, and many with substance use problems have other diagnoses. This can mean the categories overlap, that disorders share risk factors (the idea behind HiTOP's broad spectra), or that one condition raises the risk for another. Heterogeneity is the other side: two people with the same label may differ widely in symptoms, causes and response to treatment, as the counting exercise in the previous lesson showed.

Labelling and stigma

Labels have costs and benefits. Benefits: validation, access to services, community and understanding. Costs: stigma, discrimination in work and relationships, self-stigma (internalized shame), and the self-fulfilling prophecy, where expectations from others shape behavior. Sociologist Erving Goffman described how stigma spoils identity in Stigma (1963), and Thomas Scheff argued that labelling can stabilize deviance. Evidence on how labels affect outcomes is mixed, but there is clear evidence that public stigma reduces help-seeking. Person-first language, accurate education and contact with people who have recovered reduce stigma.

Common misconceptions

"Rosenhan proved psychiatry is a fraud." The study had serious limits, and its reliability is questioned. It still reminds us how powerful labels and settings are.

"High agreement means the diagnosis is real." You can reliably agree on a category that does not correspond to a distinct condition.

"Comorbidity means misdiagnosis." Overlap is expected when problems share causes or when one raises the risk of another.

Try it

The first demonstration computes Cohen's kappa for two clinicians. Change the counts to see how chance agreement changes the result. The numbers are invented for practice.

Python Cohen's kappa calculator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The second demonstration shows why percent agreement can mislead. When a disorder is rare, two raters who both say "no" nearly always agree, so agreement looks high even if they cannot spot a single case.

Python Why raw agreement misleads

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does inter-rater reliability measure?

    It is agreement between raters. Validity concerns whether the category is real and meaningful.
  2. 2. Why is Cohen's kappa preferred to raw percent agreement?

    When a disorder is rare, raters agree often by chance, so raw agreement can look high.
  3. 3. What did Rosenhan (1973) report?

    Critics later questioned the study's interpretation and details, including Spitzer (1975) and Cahalan (2019).
  4. 4. A diagnosis that clinicians agree on, but that does not predict anything about course or treatment, is

    Consistency alone does not prove the category is meaningful.
  5. 5. Which is a possible explanation for comorbidity?

    Overlap often reflects shared causes or sequential risk.

Summary

  • Reliability is agreement; validity is whether a category is real and useful. You need both.
  • Kappa adjusts agreement for chance. Some DSM-5 diagnoses showed low agreement in field trials.
  • Rosenhan's study showed the power of labels but is weak evidence, and it has been challenged.
  • Comorbidity and heterogeneity encourage dimensional views; labels bring benefits and stigma.

Next: Chapter 2 Review