Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 39 ยท Requests and APIs

Finding and Using JSON APIs

Skip HTML parsing: find the JSON endpoints a website uses, read them with requests, and extract nested data safely in Python. Runnable offline examples.

Many modern pages are not built on the server. The HTML you receive is an empty shell, and JavaScript then asks a JSON endpoint for the real data and draws it. If you can call that endpoint yourself, you get clean, structured data with no HTML parsing at all. It is faster, more stable, and kinder to the server than downloading full pages.

This lesson shows how to find such endpoints, how to read their responses, and how to dig through nested JSON without your script crashing on a missing key. The examples parse JSON held in strings, so they run anywhere.

Why JSON beats HTML when you can get it

  • The structure is explicit: numbers are numbers, lists are lists.
  • Field names rarely change when the site redesigns its layout.
  • Responses are small compared to a full page with scripts, styles and ads.
  • Pagination information is usually included (next page, total count).

Before scraping HTML, always spend five minutes looking for an API. If the site offers an official, documented API, use that and read its terms; it is the most reliable and most welcome option.

Finding the endpoint with DevTools

  1. Open the page in your browser and press F12 to open DevTools.
  2. Go to the Network tab and filter by Fetch/XHR.
  3. Reload the page (or scroll, click "Load more", change a filter).
  4. Click each request and look at the Response or Preview tab. You are looking for JSON that contains the data you see on screen.
  5. Note the Request URL, the method, the query parameters and any special headers.
  6. Right-click the request and choose Copy as cURL, or copy the URL, and reproduce it in your code.
flowchart LR
  A["Browser loads HTML shell"] --> B["JavaScript runs"]
  B --> C["Fetch /api/products?page=1"]
  C --> D["JSON response"]
  D --> E["Page draws the list"]
  F["Your scraper"] -->|"calls the same URL"| C

Your scraper can skip the first two steps and talk to the endpoint directly.

Endpoints that need a login, an API key, or that are marked private in the site's terms are not open invitations. See Legal and ethical scraping.

Reading the response

In real code the call is short (read-only):

Python
import requests

resp = requests.get(
    "https://shop.test/api/products",
    params={"page": 1, "per_page": 50},
    headers={"Accept": "application/json", "User-Agent": "DemoBot/1.0"},
    timeout=10,
)
resp.raise_for_status()
data = resp.json()

resp.json() is json.loads(resp.text) with encoding handling. It raises requests.exceptions.JSONDecodeError if the body is not JSON, which usually means you were served an HTML error or a CAPTCHA page. Check the Content-Type header when in doubt.

Here is a typical payload parsed with the standard json module:

Python Parse and walk a JSON response

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the third item: "price": null becomes None in Python. Real APIs are full of these gaps, so never assume a nested key exists.

Digging safely through nested data

A chain such as item["seller"]["address"]["city"] raises KeyError or TypeError at the first missing level. Write one small helper and use it everywhere:

Python A safe getter for nested JSON

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Libraries such as glom or jmespath do the same with richer syntax, but this ten-line helper covers most needs.

Flattening into rows

For storage you usually want flat dictionaries, one per record. A list comprehension with dig gives every row the same keys:

Python Flatten nested records into rows

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

GraphQL and POST endpoints

Some sites use one endpoint such as /graphql and send a query in a POST body. DevTools shows the Payload tab for these. You copy the JSON body, tweak the variables, and send it:

Python
query = {
    "query": "query($slug:String!){ product(slug:$slug){ name price } }",
    "variables": {"slug": "trail-shoe"},
}
resp = requests.post("https://shop.test/graphql", json=query, timeout=10)
print(resp.json()["data"]["product"])

Passing json= sets the Content-Type: application/json header and serializes for you.

Things that go wrong

  • Signed or short-lived tokens. The URL contains a token= that expires. You must fetch a fresh one from the page first.
  • Required headers. Some endpoints check Referer, X-Requested-With or a CSRF header. Copy only what is needed.
  • Rate limits. APIs are often stricter than pages. Read the X-RateLimit-* and Retry-After headers.
  • Changes without notice. Undocumented endpoints can change any day. Keep parsing code small and add checks that fail loudly when expected keys vanish.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Where in the browser DevTools do you look for the JSON requests a page makes?

    The Network tab lists every request; the Fetch/XHR filter isolates the ones that load data.
  2. 2. What does a JSON null become in Python?

    json.loads maps null to None, so check for it before indexing deeper.
  3. 3. Why use a helper like dig() instead of chained indexing?

    Chained indexing raises KeyError or TypeError at the first missing level; a safe getter returns a default.
  4. 4. What usually causes resp.json() to fail with a decode error?

    Blocked or failed requests often return HTML, which is not valid JSON.

Summary

  • Look for JSON endpoints in the Network tab before parsing HTML.
  • response.json() gives dictionaries and lists; null becomes None.
  • Navigate nested data with a safe getter and flatten records into uniform rows.
  • GraphQL and POST endpoints work the same way with json=.
  • Undocumented endpoints can break or be restricted; respect limits and terms.

Next: Pagination Patterns