Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 39 ยท Dynamic Pages and Browser Automation

Playwright Basics

Scrape JavaScript sites with Playwright for Python: sync API, locators with auto-waiting, reading content, intercepting API responses and blocking heavy resources.

Playwright is a newer browser-automation library from Microsoft. Like Selenium it controls a real browser, but it was designed with modern, JavaScript-heavy sites in mind. Its best feature for scrapers is auto-waiting: before it clicks or reads something, it automatically waits for the element to exist, be visible and be stable. You write far fewer waits and get fewer flaky runs.

It also installs its own browsers (Chromium, Firefox, WebKit), so there is no driver-matching to worry about, and it can listen to the network traffic of the page, which means you can capture the JSON the page loads instead of scraping the rendered result.

The browser itself cannot run in our in-page editor, so the Playwright code below is read-only. Runnable stand-ins cover the logic you can test without a browser, such as parsing the HTML you get back.

Install

Terminal
pip install playwright
playwright install chromium

A first script (sync API)

Python
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    context = browser.new_context(
        user_agent="DemoBot/1.0 (+https://example.org/bot)",
        viewport={"width": 1280, "height": 900},
        locale="en-US",
    )
    page = context.new_page()
    page.goto("https://example.com/", wait_until="domcontentloaded")

    print(page.title())
    print(page.locator("h1").inner_text())

    browser.close()

The structure is: browser (one process) contains contexts (isolated profiles with their own cookies, like incognito windows) which contain pages (tabs). Creating a new context is cheap, so use one per independent job.

flowchart TD
  B["Browser (Chromium)"] --> C1["Context A: cookies, user agent"]
  B --> C2["Context B: separate cookies"]
  C1 --> P1["Page / tab"]
  C1 --> P2["Page / tab"]
  C2 --> P3["Page / tab"]

Locators and auto-waiting

A locator is a lazy description of how to find elements; Playwright resolves it when you act on it and waits as needed.

Python
cards = page.locator("li.product")           # CSS
cards.first.wait_for()                       # wait for at least one
print(cards.count())

for i in range(cards.count()):
    card = cards.nth(i)
    name = card.locator("h2").inner_text()
    price = card.locator(".price").inner_text()
    href = card.locator("a").get_attribute("href")
    print(name, price, href)

page.get_by_role("button", name="Load more").click()   # waits until clickable
page.get_by_placeholder("Search").fill("trail shoes")
page.keyboard.press("Enter")

Role-based and text-based locators (get_by_role, get_by_text, get_by_label) tolerate layout changes better than long CSS chains. locator.all_inner_texts() returns a list of texts in a single call, which is much faster than a Python loop over nth(i).

Hand the HTML to BeautifulSoup

As with Selenium, the cleanest design keeps the browser for rendering and uses your normal parser for extraction:

Python
page.goto(url)
page.wait_for_selector("li.product")
html = page.content()

Then parse html with BeautifulSoup. Here is that second half, which you can run, using a saved snapshot as a stand-in for page.content():

Python Parse a rendered snapshot

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Capture the API instead of the DOM

Often the page loads its data from an endpoint (JSON APIs). With Playwright you can listen for those responses and read the JSON directly, even when the endpoint needs tokens the page generates itself:

Python
captured = []

def on_response(response):
    if "/api/products" in response.url and response.status == 200:
        captured.append(response.json())

page.on("response", on_response)
page.goto("https://shop.example/")
page.wait_for_load_state("networkidle")

products = [item for batch in captured for item in batch["items"]]

Or wait for one specific response while you trigger it:

Python
with page.expect_response(lambda r: "/api/products?page=2" in r.url) as info:
    page.get_by_role("button", name="Next").click()
data = info.value.json()

JSON from the page's own network call is the best of both worlds: the browser handles all the tokens and scripts, and you receive structured data.

Here is the processing step you would put after capturing, run on a sample response:

Python Process captured API batches

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Faster runs: block what you do not need

Images, fonts and trackers slow every page and cost the site bandwidth. Block them with a route handler:

Python
def block_heavy(route):
    if route.request.resource_type in {"image", "font", "media"}:
        route.abort()
    else:
        route.continue_()

context.route("**/*", block_heavy)

Saving login state

After a legitimate login you can save cookies and storage once and reuse it:

Python
context.storage_state(path="state.json")
# next run:
context = browser.new_context(storage_state="state.json")

Treat state.json like a password file.

Async API

For many pages in parallel use async_playwright with async/await and one context per task. Combine it with a semaphore (asynchronous scraping) so you never open more tabs than the site and your machine can handle.

Selenium or Playwright?

Selenium Playwright
Waiting Explicit waits you write Auto-waits built in
Setup Driver auto-managed Own browsers via one command
Network interception Limited First-class (route, on("response"))
Maturity and ecosystem Very large Newer, growing quickly
Typical choice for new scraping projects Fine Usually easier

Both are legitimate. Pick the one your team already knows if the project is small.

A headless browser looks like a person to simple checks, but that is no reason to ignore rules. If a site requires a login, shows a CAPTCHA, or forbids automation in its terms, stop. See Legal and ethical scraping.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does Playwright's auto-waiting do?

    Actions wait for the element to exist, be visible and stable, which removes most manual waits.
  2. 2. What is a browser context in Playwright?

    Contexts behave like separate incognito sessions and are cheap to create.
  3. 3. Why is capturing the page's own API responses attractive?

    The browser handles tokens and scripts while you read clean JSON from the network traffic.
  4. 4. How do you speed up scraping runs in Playwright by skipping images and fonts?

    context.route lets you abort unneeded resource types before they download.

Summary

  • Playwright controls real browsers with built-in auto-waiting and strong network tools.
  • Hierarchy: browser, contexts, pages; use locators for lazy, auto-waiting element access.
  • Prefer capturing the page's JSON responses, or send page.content() to BeautifulSoup.
  • Block images and fonts to speed up runs; save login state files carefully.
  • Browser automation stays a last resort and does not override a site's rules.

Next: Waits, Scrolling and Interactions