Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 36 ยท User Research

Surveys and Questionnaires

Learn how to design UX surveys that give trustworthy answers. Covers question types, Likert scales, wording pitfalls, response bias, sample size and a margin-of-error calculator.

A survey asks many people the same questions and counts the answers. It cannot explain deep motives the way an interview can, but it can tell you how common a behavior or problem is. Combined with qualitative research, it helps you decide what matters most.

When a survey is the right tool

Use a survey when you want to:

  • Measure how many users have a need, problem or preference.
  • Compare groups (new versus experienced users).
  • Track satisfaction over time.
  • Validate a pattern you saw in interviews.

Do not use a survey to discover what questions to ask. If you do not know the answer options, interview first. A survey of unknown territory produces a pile of "Other".

Question types

  • Multiple choice (single or multi-select): easy to analyze. Options should be mutually exclusive and cover all likely answers; add "Other" or "None" where needed.
  • Rating scales: "How easy was it to complete your order?" from 1 (very difficult) to 5 (very easy).
  • Ranking: order items by importance. Tiring beyond five or six items.
  • Open text: gives rich detail but is costly to analyze. Use sparingly, often one at the end ("Anything else you'd like to tell us?").
  • Demographic questions: only ask what you will use, and make them optional where sensitive.

Likert scales

A Likert scale measures agreement with a statement: strongly disagree, disagree, neutral, agree, strongly agree. Tips: label every point (not just the ends), keep the direction consistent, and consider whether you truly need a neutral midpoint. Five or seven points are common.

Writing good questions

Problem Bad example Better
Leading How great was our fast checkout? How would you rate the checkout?
Double-barreled Rate the speed and design of the app. Ask two separate questions
Vague Do you use the site often? How many times in the last month did you visit?
Jargon Did the IA meet your needs? Could you find what you were looking for?
Loaded Do you agree that our excellent support is helpful? How helpful was our support?
Absolutes Do you ALWAYS read reviews? How often do you read reviews?

Keep surveys short. Every extra question lowers completion. Put the easy and interesting ones first, sensitive ones last, and show progress. Test with a few people before sending.

Bias and response rates

  • Self-selection: people who answer are often the very happy or the very angry.
  • Acquiescence: some people tend to agree with statements.
  • Order effects: earlier questions can influence later ones.
  • Non-response: if only a small share reply, the results may not represent everyone.

Report the number of respondents and how you recruited them. Be honest about limits.

Good survey habits

Short Only what you will use
Clear One idea per question
1 / 2

Sample size and margin of error

How many responses do you need? For a simple proportion (say, "62 percent of respondents prefer option A"), the margin of error at 95 percent confidence is approximately:

1.96 ร— sqrt(p ร— (1 โˆ’ p) / n)

where p is the observed proportion and n the number of responses. With 100 responses and p = 0.62, the margin is about plus or minus 9.5 points; with 400 responses, about 4.8. This formula assumes a random sample, which real UX surveys rarely have, so treat it as an optimistic guide.

The stand-in below computes it so you can feel how slowly accuracy improves with more responses.

JS Margin of error (stand-in)

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Quadrupling the sample only halves the margin. That is why very large samples are rarely worth it for simple decisions, while small samples can mislead.

Analyzing results

  • Report distributions (how many chose each option), not just averages. A mean of 3 can hide a split between 1s and 5s.
  • Segment by group when sample sizes allow.
  • Cross-check with behavior from analytics: what people say and do can differ.
  • Read open-text answers and code them into themes, as with interview notes.
  • Say what you do not know, and avoid precise percentages from tiny samples.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. When is a survey the right choice?

    Surveys count; they do not explore. Interview first if you do not know the options.
  2. 2. Which question is double-barreled?

    It asks two things at once, so the answer is unclear.
  3. 3. What happens to the margin of error if you quadruple the sample size?

    The margin shrinks with the square root of n.
  4. 4. Why report distributions and not only averages?

    A mean of 3 may be half 1s and half 5s.

Summary

  • Use surveys to measure how many, after interviews have shown what to ask.
  • Write short, neutral, single-idea questions and label scales.
  • Beware of self-selection and other biases; report sample size honestly.
  • Margin of error shrinks slowly with sample size and assumes random sampling.
  • Analyze distributions and combine with behavioral data.

Next: Personas and Empathy Maps