Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 76 ยท Getting Data

Collecting Data from APIs and Web Pages

Collect data for analysis from JSON APIs and web pages: requests, pagination, rate limits, API keys, caching raw responses, HTML tables, and polite, legal scraping.

Sooner or later the data you need is not in a file somebody handed you. It sits behind a web service: weather readings, exchange rates, public statistics, product listings, social posts. Collecting it yourself is a core data-science skill, and it is also one where small mistakes cause large problems: lost data, banned accounts, broken analyses that nobody can repeat.

This lesson is about collecting data for analysis. It shows the shape of a reliable collection job: ask for data politely, handle pages and errors, keep the raw answers, and turn them into a clean table. The Python course covers the tools in more detail in Web Scraping and the requests library reference. Because the code on this page runs in a sandbox without network access, the examples use a fake API written in Python that behaves like a real one. The code you write against it is the code you would write against a real service.

APIs first, scraping last

There are three ways to get data from the web, in order of preference:

  1. Download a file. Many organisations publish CSV, JSON or Excel files. Simple and stable. Prefer this when it exists.
  2. Use an API. An API (application programming interface) is a web address that returns structured data, almost always JSON. It is meant for programs, it has documentation, and it usually has rules about how often you may ask.
  3. Scrape a web page. Extract data from HTML meant for humans. It breaks whenever the page layout changes, and it may be forbidden. Use it only when the first two are impossible.
flowchart TD
    A["I need data from a website"] --> B{"Is there a downloadable file?"}
    B -->|yes| C["Download it"]
    B -->|no| D{"Is there an API?"}
    D -->|yes| E["Call the API politely"]
    D -->|no| F{"Do the terms allow scraping?"}
    F -->|yes| G["Scrape carefully, low rate"]
    F -->|no| H["Ask for permission or use another source"]

Choose the most stable, most legitimate source first.

Calling an API: the shape of a request

A request has a URL (the address), a method (GET to read), optional query parameters (filters, such as ?city=Oslo&days=3), and headers (extra information, such as an API key). The server answers with a status code (200 means success) and a body, usually JSON.

On your own computer, with the requests library (python -m pip install requests), a call looks like this. This snippet is read-only because the browser sandbox has no open network:

Python
import requests

response = requests.get(
    "https://api.example.com/v1/readings",
    params={"city": "Oslo", "days": 3},
    headers={"Authorization": f"Bearer {API_KEY}"},
    timeout=10,                       # always set a timeout
)
response.raise_for_status()           # raises an error for 4xx and 5xx answers
data = response.json()                # JSON text -> Python dict/list

Four habits in those lines matter:

  • timeout: without it a program can wait forever for a server that never answers.
  • raise_for_status(): turns "404 not found" or "500 server error" into an exception instead of silently parsing an error page as data.
  • params=: let the library encode query strings; never build URLs by gluing strings.
  • Secrets in the environment: keep the API key in an environment variable (os.environ["API_KEY"]), never in your code or your repository.

From JSON to a DataFrame

APIs return nested JSON: dictionaries inside lists inside dictionaries. Your job is to flatten it into a table. pd.json_normalize does most of the work:

Python Flatten an API response into a table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The nested wind object became two columns, wind_speed and wind_dir, and the missing direction (JSON null) became a missing value. See JSON and Nested Data for deeper nesting and explode.

Pagination: getting all the pages

APIs rarely return everything at once. They return a page of results (for example 100 items) and tell you how to get the next one: a page number, an offset, or a cursor (an opaque token for "continue from here"). Your code must loop until there is no next page.

The fake API below returns three items per page with a cursor. The loop is the same for a real service:

Python Following a cursor through all pages

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Pitfalls: check that you really got everything (compare the row count to a total field if the API provides one), keep your loop from running forever (add a maximum page count), and check that IDs are unique, because some APIs repeat items across pages when new data arrives while you are paging.

Be a good client: rate limits, retries and caching

Servers protect themselves. They limit how many requests you may send per minute and answer with status 429 (too many requests). Overloaded servers answer with 5xx errors. A reliable collector copes with both:

  • Pause between requests. Read the documentation for the limit and stay well under it.
  • Retry with backoff. After a 429 or a 5xx, wait, then try again, doubling the wait each time (1 s, 2 s, 4 s), and give up after a few attempts. Many servers send a Retry-After header that tells you how long to wait.
  • Do not retry client errors. A 400 or 404 will fail again; fix the request instead.
  • Cache. Store every raw response on disk the first time you fetch it, and read the file next time. Your analysis becomes repeatable, you do not hit the server again, and you keep the raw data if the source changes or disappears.
Python Retry with exponential backoff

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The small random addition (jitter) stops many clients from retrying at exactly the same moment. Real code replaces the commented line with time.sleep(delay).

Scraping web pages

When there is no API, you can read the page's HTML. The process is: fetch the HTML, parse it into a tree, select the elements you want, and extract text and attributes. Pages often contain tables, and pandas can read those directly with pd.read_html, which returns one DataFrame per table:

Python Reading an HTML table and parsing with BeautifulSoup

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

read_html guessed the numeric types, so the temperature column is already float64. BeautifulSoup handles whatever is not a table: select with CSS selectors ("ul.links a"), read text with get_text() and attributes with a["href"]. On your own machine you would get the HTML with requests.get(url, timeout=10).text. Also note that pages built with JavaScript do not contain their data in the HTML you download; the data usually arrives from a hidden JSON API, which you can often find in the browser's Network tab and call directly. That is better than scraping.

Treat scraping as a privilege:

  • Read the site's terms of service and its robots.txt file, which lists which paths automated clients may visit.
  • Send a clear User-Agent with a contact address, so that the site owner can reach you.
  • Go slowly: one request per second or slower, and only what you need.
  • Do not collect personal data without a lawful reason. Privacy laws (GDPR and others) apply even to public pages.
  • Do not republish copyrighted content. Facts are one thing; a database or article is another.

Python's standard library can read robots.txt rules for you:

Python Checking robots.txt rules

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Not being blocked by robots.txt does not make scraping legal; it is only one signal. When a project depends on scraped data, ask someone qualified. See also Data Sources for the broader picture.

A checklist for a collection job

  1. Is there a file or an API? Read the documentation, including limits and licence.
  2. Authenticate safely (environment variables, never in code).
  3. Set a timeout, check the status, and handle pagination to the very end.
  4. Pause between requests, retry only transient errors, back off.
  5. Save the raw responses with a timestamp, then parse from the saved files.
  6. Validate: expected columns, types, row counts, duplicates (see Testing Data Code).
  7. Record where, when and how the data was collected, so that the analysis can be repeated.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why call response.raise_for_status() after a request?

    Without it, a failed request can return an HTML error page that you might parse as if it were data.
  2. 2. Which HTTP errors are normally worth retrying with a delay?

    429 and 5xx errors are usually temporary. 4xx errors caused by your request will fail again, so retrying them wastes calls.
  3. 3. Why should you save raw API responses to disk?

    Raw responses are your evidence. They let you reparse, audit and reproduce results without calling the API again.
  4. 4. A JavaScript-heavy page shows data but your downloaded HTML has none. What is the best next step?

    The data usually arrives from a hidden JSON endpoint. Using it is simpler and more stable than scraping the rendered page.
  5. 5. Where should an API key live?

    Keys in code or in URLs leak through version control and logs. Environment variables keep them out of the code.

Summary

  • Prefer a downloadable file, then an API, and scrape web pages only as a last resort, within the site's terms.
  • Every request needs a timeout, a status check (raise_for_status) and parameters passed via params=; keep keys in environment variables.
  • Flatten JSON with pd.json_normalize; loop through pages or cursors until the end and check counts and unique IDs.
  • Respect rate limits, retry only 429 and 5xx errors with exponential backoff and jitter, and cache raw responses.
  • pd.read_html reads tables; BeautifulSoup selects other elements with CSS selectors.
  • Read robots.txt and the terms of service, identify yourself and go slowly.

Next: Loading Data from CSV, JSON and SQL shows the next step: getting the collected data into pandas.