Skip to content
Tutorials Center
My learning

Topics

Lesson 10 of 76 ยท Getting Data

Where Data Comes From

Survey where analysis data comes from: files, databases, APIs, logs, surveys and experiments, plus scraping basics, public datasets and data contracts.

Every dataset was created by someone, for some purpose, in some way. That origin shapes what the data can and cannot tell you. A table of purchases was made to run a business, not to answer your research question. A survey was written by people with their own wording and assumptions. If you do not know how a number came to exist, you cannot judge whether to trust it.

This lesson tours the common sources, tells you what to ask about each, and ends with a small hands-on example that reads a CSV and a JSON payload. The next lesson, Loading Data from CSV, JSON and SQL, goes deeper into the code.

The main sources

flowchart LR
    F["Files<br/>CSV, Excel, Parquet"] --> P["Your analysis"]
    DB["Databases<br/>SQL tables"] --> P
    API["APIs<br/>JSON over HTTP"] --> P
    L["Logs and events<br/>clicks, sensors"] --> P
    S["Surveys"] --> P
    E["Experiments"] --> P
    W["Web pages<br/>scraping"] --> P

Seven common routes by which data reaches an analysis.

Files

CSV, Excel, JSON and Parquet files are the simplest way to share data. They are easy to start with, but a file is a snapshot: it does not tell you when it was exported, by whom, or with which filters. Always record the file's name, date and origin. CSV is plain text and universal but has no types; Parquet stores types and compresses well, which is why it is popular for large tables.

Databases

Most company data lives in relational databases and is read with SQL. Databases give you consistency and the ability to filter and join before data reaches your computer. Ask which tables are the source of truth, how often they refresh, and whether you are reading from a replica built for analysis rather than the live system. If SQL is new to you, the SQL tutorial covers it from scratch.

APIs

An API (application programming interface) is a service your code can call to fetch data, often returning JSON. APIs give fresh data and structured fields, but they impose limits: rate limits, pagination, authentication keys, and the right of the provider to change fields without notice. Keep keys out of your code and out of version control.

Logs and event streams

Applications and devices record what happens: page views, clicks, sensor readings, errors. Logs are huge and detailed, and they record behaviour rather than opinions. They are also messy: duplicated events, missing events (ad blockers, offline devices), clocks that disagree. Logs usually need aggregating before analysis.

Surveys

Surveys ask people directly, so they are the only way to learn attitudes and reasons. They come with their own problems: question wording, who chooses to answer, people giving the answer they think is expected. See Sampling and Collection Bias.

Experiments

When you randomly assign people to treatments (for example two versions of a page), you create data designed to answer causal questions. It is the strongest kind of evidence but costs time and needs careful planning. See A/B Testing and Experiments.

Web scraping

Scraping means extracting data from web pages when no API exists. It is a last resort and a legal and ethical minefield, covered next.

Scraping basics and legality

Treat scraping carefully:

  • Check for an API or a download first. Many sites offer one.
  • Read the terms of service and robots.txt. Terms may forbid automated access, and robots.txt states which paths the site asks bots to avoid.
  • Be gentle. Send few requests, pause between them, and identify your program honestly.
  • Think about the data, not just access. Copyright can apply to content; personal data is protected by laws such as the GDPR in Europe. Public does not mean free to use for any purpose.
  • Never bypass logins, paywalls or technical protections.

The rules differ between countries and change over time. This is general guidance, not legal advice: when a project depends on scraped data, ask someone qualified. Because the browser sandbox here has no network, scraping examples in this course are read-only.

Public and open datasets

Many governments, universities and organizations publish open data: census tables, weather records, transport timetables, health statistics. They are excellent for practice and for context in real projects. Before using one, check:

  • Licence: can you use it commercially, and must you credit the source?
  • Documentation: is there a data dictionary explaining each column?
  • Date and coverage: what period and population does it cover?
  • Method: how was it collected, and what was left out?

In this course we mostly generate synthetic data so you can see exactly what is true, but the questions above apply to every real dataset.

Data contracts

A data contract is an agreement between the people who produce a dataset and the people who consume it. It states the columns, their types, units, allowed values, how often the data updates, and who to call when it breaks. Without one, a producer renames a column on Friday and Monday's report fails silently.

Even without formal tooling you can write a lightweight contract as a short checklist or a few assertions in code. A contract might say:

TEXT
Table: orders (one row per order)
order_id    integer, unique, never null
placed_at   timestamp in UTC
total_eur   decimal >= 0, tax included
status      one of: paid, refunded, cancelled
Refresh: daily by 06:00 UTC. Owner: payments team.

Testing Data Code and Data Validation shows how to turn such a contract into automatic checks.

Questions to ask about any source

  1. Who produced it, and why?
  2. What does one row represent (the grain)?
  3. When was it collected, and how fresh is it?
  4. Who or what is missing? Is every person or event recorded?
  5. How was each field measured, and in what unit?
  6. Has the definition changed over time?
  7. Am I allowed to use it for this purpose?

A small hands-on example

The browser has no files or network, so we simulate two sources with strings: a CSV export and a JSON response such as an API might return. Reading them shows the typical difference: CSV is flat text, JSON can nest.

Python Reading a CSV export and a JSON response

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that json_normalize turned the nested customer object into the flat columns customer.country and customer.tier. Now the two sources share an order_id, and you can combine them:

Python Combining two sources and checking the result

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Order 4 has no matching customer record, so a left join keeps it with a missing country. That is exactly the kind of gap you want to notice and ask the data owner about. The validate="one_to_one" argument makes pandas raise an error if the join keys are not unique, which guards against the double counting you saw in Types of Data and Measurement.

Warning: Never paste passwords, API keys or personal data into code that you share, commit or run in a shared notebook. Load secrets from environment variables or a secrets manager.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which source is best suited to answering a causal question such as 'does the new page increase sign-ups'?

    Randomly assigning users to versions creates comparable groups, which is what allows a causal conclusion.
  2. 2. What is a data contract?

    A data contract describes what a dataset promises, so changes do not silently break the people who depend on it.
  3. 3. Before scraping a website, what should you do first?

    An official API or download is easier and safer. Terms and robots.txt tell you what the site allows.
  4. 4. Why use validate='one_to_one' in a pandas merge?

    validate checks the relationship you expect between the tables, catching duplicate keys that would otherwise duplicate rows.

Summary

  • Data comes from files, databases, APIs, logs, surveys, experiments and scraping; each has typical strengths and traps.
  • Always ask who produced the data, what a row means, when it was collected and who is missing.
  • Scrape only as a last resort, politely and within the terms, the law and the privacy of the people involved.
  • Check licence, documentation and coverage of public datasets.
  • Write a data contract so changes in the source do not break your analysis silently.

Next: Loading Data from CSV, JSON and SQL, the practical code for reading each source.