Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 36 ยท Usability Testing

Analyzing Findings and Severity

Turn raw test notes into prioritized, actionable findings with affinity mapping, severity ratings and a clear report, using a runnable severity scoring example.

After the last session you have hours of recordings and pages of notes. The value of the test is created in the next step: turning that pile into a short list of prioritized problems, each with evidence and a proposed fix. A test without analysis changes nothing.

Step 1: Gather observations

Collect all observations from every moderator and note-taker. Each one should be a single fact or quote ("P3 did not find the Returns link in the header and scrolled to the footer"), not an opinion ("navigation is bad"). Remove duplicates and keep the participant ID, task and a timestamp so you can go back to the recording.

Step 2: Group with affinity mapping

Write each observation on a sticky note (physical or digital) and cluster similar ones on a wall. Name each cluster as a problem statement: "Users cannot find the return option". Do it as a team so different views surface, and group by the underlying issue, not by the screen. Clusters that hold notes from many participants point to real problems; a single note may be an outlier worth watching.

flowchart LR
  A["Raw notes<br/>and recordings"] --> B["One fact<br/>per note"]
  B --> C["Cluster into<br/>problems"]
  C --> D["Rate severity"]
  D --> E["Recommend fixes<br/>and report"]

Figure: from raw notes to a prioritized list.

Step 3: Rate severity

You cannot fix everything at once. A severity rating orders problems by how much they hurt. A widely used scale (after Jakob Nielsen) has five levels:

Rating Meaning
0 Not a usability problem
1 Cosmetic; fix if time allows
2 Minor; low priority
3 Major; important to fix, high priority
4 Catastrophe; must fix before release

Three factors guide the rating: frequency (how many users hit it), impact (how badly it blocks or confuses), and persistence (does it keep happening, or do people learn to work around it). A problem that stops purchases for 4 of 5 participants is severe; a mislabeled icon that one person noticed and quickly understood is not.

Python Score and rank problems

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The weights here are invented to make the idea concrete. Real teams use the simple 0 to 4 scale or a frequency-by-impact grid and then agree on ratings by discussion, ideally with two raters who compare and reconcile.

Step 4: Quantify (lightly)

With small samples, report counts, not percentages ("4 of 5 could not complete the task"). Task success rates, time on task and satisfaction scores are useful, but with few participants they are indicators, not proof. See UX metrics for how to compute them and show their uncertainty.

Step 5: Recommend solutions

For each high-severity problem, suggest a fix and a reason: "Add Returns to the account menu and order details page, because 3 of 4 who failed looked in Orders." Distinguish the problem (what users experienced) from the solution (one way to fix it); the team may choose a better one. Also report what worked well so those parts are not broken by redesign.

Step 6: Report so that people act

Different audiences need different reports:

  • Executives: a one-page summary: top 3 to 5 issues, impact on goals, and a recommendation.
  • Product and design team: a prioritized list with evidence, screenshots, severity and suggested fixes, plus video clips of key moments.
  • Everyone: a short highlight reel (1 to 3 minutes of clips) is far more convincing than a slide of text.

A good finding has a title, severity, evidence (who and how many, a quote, a clip), the impact on the user and the business, and a recommendation.

Invite stakeholders to watch at least one live session. People who saw a participant struggle fix problems without needing to be persuaded.

Step 7: Follow up

Add findings to the team backlog with owners, retest after changes, and track whether the severity-3 and -4 issues are closed. Learning is only complete when the design improves.

Common mistakes

  • Reporting opinions or single-participant anecdotes as facts.
  • Listing 40 equal-weight issues with no priority.
  • Blaming the user instead of the design ("people should read more carefully").
  • Only reporting negatives, so the team loses what works.
  • Skipping a retest.

Check your understanding

Quick check ยท 3 questions

of 3 answered You scored / 3 ยท perfect!

  1. 1. What is affinity mapping?

    Grouping notes by underlying issue reveals recurring problems.
  2. 2. Which factors guide severity?

    How many are affected, how badly, and whether it keeps happening determine severity.
  3. 3. With only five participants, how should you report results?

    Small samples call for counts and evidence, not false precision.

Summary

  • Turn raw notes into single-fact observations, cluster them, and name the problems.
  • Rate severity by frequency, impact and persistence, using a simple 0 to 4 scale.
  • Recommend fixes, report what works, and tailor the report to the audience.
  • Follow up: backlog items, owners and a retest.

Next: Heuristic Evaluation and Cognitive Walkthrough