Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 39 ยท Dynamic Pages and Browser Automation

JavaScript-Rendered Pages

Why requests sees an empty page, how to check for JavaScript rendering, and how to extract data embedded in script tags, JSON-LD and window state with Python.

You write a perfect scraper. It fetches the page, parses it, and finds nothing. You open the same URL in your browser and the product list is right there. What happened?

The browser ran JavaScript. The HTML your scraper downloaded was only a shell, and the scripts inside it built the visible page afterwards. A plain HTTP client does not run scripts, so it never sees that content. This is the single most common reason a first scraper "doesn't work", and it has a few well-known remedies. Reaching for a full browser should be the last one, not the first.

Confirm the diagnosis

Compare two views of the page:

  1. View source (Ctrl+U). This is exactly what requests receives. Search for a piece of text you can see on the page.
  2. Inspect element (F12, Elements tab). This is the page after JavaScript ran.

If the text is in the first view, ordinary parsing works. If it appears only in the second, the page is rendered by script. A quick way to be sure in code is to test for the data in the raw response before you write any parser:

Python Is the data in the raw HTML?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Your options, in order of preference

flowchart TD
  A["Page content missing from raw HTML"] --> B{"Data in a script tag<br/>or JSON-LD?"}
  B -- "yes" --> C["Parse the embedded JSON"]
  B -- "no" --> D{"JSON endpoint in<br/>Network tab?"}
  D -- "yes" --> E["Call the API directly"]
  D -- "no" --> F["Use a real browser<br/>(Playwright or Selenium)"]

Cheapest first: embedded data, then API, then browser automation.

The API route is covered in JSON APIs. Browsers come in Selenium and Playwright. Here we handle the first branch.

Data embedded in script tags

To make the first paint fast, many frameworks embed the initial data in the HTML itself. Look for these in view-source:

  • <script id="__NEXT_DATA__" type="application/json"> (Next.js)
  • <script type="application/ld+json"> (structured data for search engines)
  • window.__INITIAL_STATE__ = {...}; or window.__NUXT__ = ... assigned inside a script
  • Plain <script type="application/json"> blocks with any id

Each contains the same data the page renders, already structured. Reading it is far more stable than scraping the final DOM.

Python Extract __NEXT_DATA__ and JSON-LD

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

JSON-LD deserves special attention: products, recipes, articles, events and job postings very often publish clean structured data there, because search engines read it.

State assigned in JavaScript

Sometimes the data sits inside a script as a JavaScript assignment rather than JSON. Cut the object out with a regular expression and parse it with json.loads. This works when the object is valid JSON (double-quoted keys):

Python Pull window.__INITIAL_STATE__ out of a script

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The non-greedy .*? plus the ; and newline anchors is a heuristic. When objects contain nested semicolons or the code uses JavaScript-only syntax (single quotes, trailing commas, undefined), a regex is fragile. A more reliable trick is to find the opening brace and use json.JSONDecoder().raw_decode, which parses one JSON value from a position and tells you where it ended:

Python raw_decode: parse JSON from the middle of a string

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Other hidden sources

  • Inline JavaScript variables such as var products = [...]. Same technique.
  • HTML comments that contain markup the page unhides later.
  • data-* attributes, e.g. <div data-product='{"id":5}'>, which are already in the raw HTML and parse as JSON.
  • Data in <template> tags or hidden inputs.
  • An alternative version of the page: AMP pages, mobile sites (m.example.com), or ?output=json style parameters sometimes exist. Check, but only use what the site makes publicly available.

When none of that works

If the data is generated by script from a request signed at runtime, or the page needs interaction (click to reveal, scroll to load), you need something that runs JavaScript. That means a browser controlled from code, which is slower (seconds per page instead of milliseconds), heavier (hundreds of MB of RAM) and easier to detect. It is a legitimate tool when nothing else works, and the next lessons teach it properly.

Do not use browser automation to get around a site's blocking measures. CAPTCHAs, bot walls and login gates are the site saying "no automated access". See Legal and ethical scraping.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why can requests miss content that your browser shows?

    requests only downloads the HTML; scripts that build the page never run.
  2. 2. Which view shows what requests actually receives?

    View source shows the original HTML before JavaScript changes it, which is what an HTTP client sees.
  3. 3. What is the best first thing to try when the data is not in the visible HTML?

    Embedded data and JSON endpoints are cheaper, faster and more stable than driving a browser.
  4. 4. Why use json.JSONDecoder().raw_decode on script text?

    raw_decode lets you parse an object embedded in surrounding code without a fragile regex for the closing brace.

Summary

  • Compare view-source with the inspected DOM to learn whether a page is rendered by JavaScript.
  • Prefer embedded data (__NEXT_DATA__, JSON-LD, state assigned in scripts) and JSON endpoints.
  • Use json.loads on script contents; use raw_decode when JSON is embedded in code.
  • Browser automation is the last resort because it is slow, heavy and easy to detect.

Next: Selenium Basics