Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 76 ยท What Data Science Is

Framing a Good Question

Turn a vague business request into a testable data question: choose metrics, write hypotheses, set success criteria and baselines, and know when data cannot help.

Most failed data projects did not fail because of a bad algorithm. They failed because the team answered a question nobody had asked. "Help us reduce churn" sounds like a task, but it hides a dozen decisions: what counts as churn, over what period, which customers matter, what would we do differently if we knew, and how will we know that it worked?

Framing is the skill of turning a fuzzy goal into a precise question that data can answer. It takes an hour or two, and it saves weeks. This lesson walks through the steps and ends with a worksheet you can reuse.

From business question to data question

Start from the decision, not from the data. Ask: what will someone do differently depending on the answer? If nothing changes, the analysis is a curiosity.

flowchart LR
    B["Business goal<br/>reduce cancellations"] --> D["Decision<br/>who gets a retention offer?"]
    D --> Q["Data question<br/>which active customers will cancel within 30 days?"]
    Q --> M["Metric<br/>recall among the top 10% riskiest"]
    M --> S["Success criterion<br/>beats the current rule by 20%"]

A business goal becomes a decision, then a data question, then a measurable success criterion.

Compare a weak and a strong version of the same request:

Weak Strong
Understand our customers better Which of the three customer groups we already target responds best to a discount?
Predict sales Forecast weekly units per store for the next 8 weeks, so purchasing can order stock
Is our app good? Did the redesigned checkout raise the share of visitors who complete a purchase?

The strong versions name a population, a time frame, a quantity and a use.

Choosing metrics

A metric is a number computed from data that stands for something you care about. Choosing it is a modelling decision, not a detail. Three common traps:

  • Proxy drift. "Clicks" is easy to measure but may not mean customers are happy. A metric that is easy to move is often easy to game.
  • Ratios hiding sizes. A conversion rate of 50% from 2 visitors is not the same as 50% from 20,000.
  • Ambiguous definitions. "Active user" must be defined: logged in once in 30 days? Made a purchase? Say it exactly and write it as code.

Writing the definition as a function forces precision. Here, "active customer" is defined in code, and you can see how much the definition changes the answer:

Python The same question, three definitions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The same customers are "mostly active" or "mostly inactive" depending on the rule. Neither is wrong, but every chart and report must say which one it uses. Agree on a definition before you start counting.

Hypotheses and success criteria

A hypothesis is a specific claim that data could contradict. "Customers like the new design" cannot be wrong. "The new design raises checkout completion by at least one percentage point" can.

Write three things down before analyzing:

  1. The claim: what you expect and in which direction.
  2. The metric and population: what is measured, for whom, over what time.
  3. The success criterion: the result that would make you act, and the result that would make you stop.

Setting the threshold in advance protects you from fooling yourself. If you only decide what counts as "good" after seeing the numbers, almost any result can be called a success. Hypothesis Testing Done Right and A/B Testing and Experiments build on this idea.

Baselines: what are you comparing to?

A result only means something against a reference. A model with 90% accuracy sounds great until you learn that 92% of customers never cancel, so "predict nobody cancels" is already 92% accurate. Always define a baseline:

  • the current process (the rule people use today);
  • a naive rule (predict the average, predict last week's value, predict the most common class);
  • a simple model that a colleague could explain in a minute.
Python A naive baseline can look impressive

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The do-nothing rule scores high accuracy and catches zero cancellations. That is why the metric must match the goal: here, catching cancellations (recall) matters, not raw accuracy. Evaluating Models with the Right Metrics goes deeper.

Scoping

A good scope fits the time, data and decision. Shrink it until you can finish a first answer in days, not months:

  • One population first (new customers in one country), then widen.
  • One decision the result feeds.
  • One time horizon (next 30 days, next quarter).
  • A list of what is explicitly out of scope, so nobody expects it.

When data cannot answer the question

Sometimes the honest answer is "this data cannot tell us". Watch for these signs:

  • The outcome was never recorded. You cannot predict satisfaction if nobody measured it.
  • Too little data, or too few events of the kind you care about.
  • Confounded by design. If only the best stores got the new display, you cannot separate the display from the store.
  • The decision is already made, and the analysis is wanted to justify it.
  • The data is not allowed to be used for this purpose (privacy, consent).

In these cases, say so early and propose a way forward: collect new data, run a small experiment, use a proxy with its limits stated, or answer a narrower question. A clear "not yet" is a valuable result.

A framing worksheet

Copy this into your project notes and fill it in before you open any data:

TEXT
Business goal:
Decision this will change, and who makes it:
Data question (one sentence, with population and time frame):
Type of question (descriptive / predictive / causal):
Primary metric (exact definition):
Baseline to beat:
Success criterion (set before looking at results):
Data needed and where it lives:
Known risks (bias, privacy, missing outcome):
Out of scope:
Deadline and first deliverable:

If you cannot fill in a line, that line is the next thing to find out.

Tip: Say the data question aloud to a non-technical colleague. If they cannot repeat it back, or if two people understand it differently, it is not framed yet.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What should you establish first when a stakeholder says 'Help us reduce churn'?

    Framing starts from the decision and the definition of the outcome. Tools come later.
  2. 2. Why set the success criterion before looking at results?

    If the threshold is chosen after the fact, almost any outcome can be described as a success. A pre-set criterion keeps the analysis honest.
  3. 3. 92% of customers never cancel. A rule that predicts 'nobody cancels' gets 92% accuracy. What does this show?

    With imbalanced classes a trivial rule scores high accuracy while catching nothing. Choose metrics such as recall and always compare to a baseline.
  4. 4. Which is a sign that data cannot answer your question?

    Without a recorded outcome, you cannot measure or predict it. You would need to collect new data or settle for a proxy.
  5. 5. Which is the strongest data question?

    It names a population, a time frame, a measurable outcome and a use. The others are goals or methods, not questions.

Summary

  • Start from the decision the answer will change, then write a precise data question.
  • Define metrics exactly, as code if possible; different definitions can give very different answers.
  • Write a falsifiable hypothesis and a success criterion before looking at results.
  • Always compare with a baseline, such as the current process or a naive rule.
  • Scope small, and say early when the data cannot answer the question.

Next: Types of Data and Measurement, a look at what kinds of values your columns hold.