Skip to content
Tutorials Center
My learning

Topics

Lesson 125 of 149 ยท Data and the Web

Browser Automation with Selenium and Playwright

Control a real browser from Python with Selenium and Playwright. Learn locators, waits, forms and the page object pattern, and practice selectors and waiting live.

In the web scraping lesson you downloaded HTML with requests and picked it apart with Beautiful Soup. That works for pages whose content arrives in the HTML. Many modern sites, though, send an almost empty page and build the content with JavaScript, hide data behind a login form, or need a click before anything appears. For those, and for testing your own web application, you need a real browser that your program can drive: it opens pages, clicks buttons, types into fields, waits for things to appear and reads the result.

Two libraries dominate this job in Python. Selenium is the long-established standard and speaks to Chrome, Firefox, Edge and Safari. Playwright is newer, ships its own browser builds, and waits for elements automatically. In this lesson you'll learn what both can do, the ideas they share (locators, waits, actions, page objects), and you will practice the parts that don't need a browser: writing good selectors, writing correct waits and structuring test code.

What runs here and what doesn't. A browser can't be installed or started from the in-browser Python editors, so the Selenium and Playwright calls below are read-only: run them on your own computer. The live examples practice the transferable skills with plain Python and Beautiful Soup, which uses the same CSS selectors as both libraries, and with a simulated clock for waiting.

Setting up

On your computer, inside a virtual environment:

Terminal
python -m pip install selenium
# Selenium 4.6+ downloads a matching browser driver for you automatically.

python -m pip install playwright
python -m playwright install chromium     # downloads the browser build Playwright uses

You also need the browser itself for Selenium (Chrome or Firefox). Playwright installs its own copy of Chromium, Firefox and WebKit with the second command.

A first script in each library

Both scripts open a page, search a form, wait for results and print text.

Python
# Selenium
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

options = webdriver.ChromeOptions()
options.add_argument("--headless=new")          # no visible window
driver = webdriver.Chrome(options=options)
try:
    driver.get("http://localhost:8000/search")
    driver.find_element(By.CSS_SELECTOR, "input[name=q]").send_keys("python")
    driver.find_element(By.CSS_SELECTOR, "button[type=submit]").click()

    results = WebDriverWait(driver, timeout=10).until(
        EC.visibility_of_element_located((By.CSS_SELECTOR, "#results li"))
    )
    print(results.text)
finally:
    driver.quit()                               # always close the browser
Python
# Playwright (synchronous API)
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("http://localhost:8000/search")
    page.locator("input[name=q]").fill("python")
    page.get_by_role("button", name="Search").click()

    print(page.locator("#results li").first.inner_text())   # waits automatically
    browser.close()

The shape is the same: start a browser, navigate, locate an element, act on it, wait for the page to change, read the result, close the browser. The most visible difference is waiting. Selenium finds elements immediately and fails if they aren't there yet, so you add explicit waits. Playwright's actions and assertions wait automatically (up to a timeout, 30 seconds by default) until the element exists, is visible and can receive the action.

Always release the browser: driver.quit() in a finally block for Selenium, or Playwright's with block. A crashed script that leaves browsers behind eats memory fast.

Locators: finding the right element

A locator says which element you mean. Your choice decides how often the script breaks when the page changes. From most to least stable:

  1. Test attributes such as data-testid="login-button". They exist only for automation and don't change when the design does. If you own the site, add them.
  2. Accessible roles and labels, like Playwright's get_by_role("button", name="Sign in") or get_by_label("Email"). They match what users perceive.
  3. IDs and names: #email, input[name=email]. Fine, as long as IDs are not generated.
  4. Meaningful CSS classes such as .error-message.
  5. Layout-dependent selectors such as div > div:nth-child(3) > span, or long XPath copied from the browser tools. These break at the next redesign. Avoid them.

Selenium's By strategies are ID, NAME, CLASS_NAME, TAG_NAME, LINK_TEXT, CSS_SELECTOR and XPATH; CSS selectors cover almost everything you need. find_element returns the first match (or raises NoSuchElementException), and find_elements returns a list that may be empty.

Because the same CSS selectors work in Beautiful Soup, you can practice them without a browser:

Python Practice CSS selectors

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice button.btn finding two buttons, which is a common source of "it clicked the wrong thing" bugs, and p.error matching even though the paragraph is hidden. A real browser distinguishes present in the page from visible, and both libraries expose that difference (is_displayed() in Selenium, is_visible() in Playwright).

Acting on the page

The usual actions are the same in both libraries:

Action Selenium Playwright
Open a URL driver.get(url) page.goto(url)
Type into a field element.send_keys("text") locator.fill("text")
Click element.click() locator.click()
Choose in a <select> Select(element).select_by_value("pro") locator.select_option("pro")
Read visible text element.text locator.inner_text()
Read an attribute element.get_attribute("href") locator.get_attribute("href")
Run JavaScript driver.execute_script("return document.title") page.evaluate("document.title")
Screenshot driver.save_screenshot("a.png") page.screenshot(path="a.png")

Selenium's send_keys appends to what is already in the field, whereas Playwright's fill clears it first. For special keys, use Keys.ENTER in Selenium (from selenium.webdriver.common.keys import Keys) and press("Enter") in Playwright.

Waiting: the number one source of flaky scripts

After a click, the page doesn't change instantly: a request goes out, JavaScript runs, elements appear. If your script looks too early, it fails randomly, depending on network speed. The tempting fix, time.sleep(5), is wrong in both directions: too short on a slow day, wasteful on a fast one. The right fix is to wait for a condition, polling until it is true or a timeout expires.

Here is what such a helper does, run against a simulated clock so the example is instant and deterministic:

Python Build an explicit wait

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

That wait_until is the heart of Selenium's WebDriverWait(driver, 10).until(...), with conditions from expected_conditions such as presence_of_element_located, visibility_of_element_located, element_to_be_clickable, text_to_be_present_in_element and url_contains. Playwright builds the same polling into every action, and gives you explicit waits when you need them: locator.wait_for(), page.wait_for_url("**/dashboard"), and expect(locator).to_be_visible() for checks. Avoid mixing Selenium's implicit wait (driver.implicitly_wait) with explicit waits, because the timeouts interact in confusing ways.

Dynamic pages, frames and tabs

  • Infinite scroll or "load more": click or scroll in a loop, and after each step wait for the item count to increase; stop when it stops growing. Playwright: page.mouse.wheel(0, 2000); Selenium: driver.execute_script("window.scrollTo(0, document.body.scrollHeight)").
  • iframes: elements inside a frame are invisible to normal locators. Selenium needs driver.switch_to.frame(element) (and default_content() to leave); Playwright uses page.frame_locator("#payment").locator(...).
  • New tabs and popups: Selenium switches with driver.switch_to.window(handle); Playwright uses with page.expect_popup() as popup:.
  • Dialogs (alert, confirm): Selenium driver.switch_to.alert.accept(); Playwright page.on("dialog", lambda d: d.accept()).
  • Hidden JSON: before automating a browser to scrape, open the browser's Network tab. If the page loads its data from a JSON endpoint, calling that endpoint with requests is faster and sturdier than driving a browser.

Structuring scripts: the page object pattern

A script that mixes selectors with business steps (find_element(...) everywhere) becomes impossible to maintain: when the login form changes, you edit twenty places. The page object pattern puts the selectors and low-level steps for one page in one class, and your tests call meaningful methods like login(user, password):

Python Page objects with a fake browser

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With a real browser you would replace FakeBrowser with an object wrapping Selenium or Playwright, and nothing in the tests would change. This is also how you combine automation with pytest: fixtures create and close the browser, and page objects keep each test a few readable lines. For Playwright there's an official plugin, pytest-playwright, that provides a ready-made page fixture.

Ethics, rules and robustness

  • Automate your own sites and tests first. Browser automation is the standard way to test web apps end to end.
  • Respect other sites. Check the terms of service and robots.txt, keep the request rate low, identify yourself where appropriate, and use an official API when one exists.
  • Don't bypass protections. Don't script around CAPTCHAs, rate limits or logins that you aren't entitled to use.
  • Keep secrets out of scripts. Read passwords from environment variables with os.environ, never from source code.
  • Capture evidence. Save a screenshot and the page HTML when a step fails; debugging a script that ran on a headless machine is nearly impossible without them. Playwright's trace viewer (context.tracing.start(...)) records a replayable timeline.
  • Selenium vs Playwright: choose Selenium if you need real Safari, an existing Selenium grid or very broad ecosystem support; choose Playwright for new projects that want auto-waiting, fast parallel contexts, tracing and a code generator (python -m playwright codegen URL records your clicks and prints the script).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why is time.sleep(5) after a click a poor way to wait for results?

    A fixed delay doesn't depend on the page. Waiting for a condition with a timeout continues as soon as the page is ready and fails clearly if it never is.
  2. 2. Which locator is usually the most stable choice if you own the website?

    Test attributes exist only for automation, so redesigns and text edits don't break them. Position-based selectors break whenever the layout changes.
  3. 3. What is the difference between find_element and find_elements in Selenium?

    The singular form fails when nothing matches. The plural returns an empty list, which makes it handy for checking that an element is absent.
  4. 4. What does the page object pattern achieve?

    Page objects separate how to find and use elements from what a test wants to check. When the page changes you update one class.
  5. 5. A page loads its list through a JSON request visible in the browser's Network tab. What is usually the best approach?

    Calling the data endpoint directly is faster, uses fewer resources and gives you clean JSON. Use a real browser when the content truly needs one, or when you are testing the interface itself.

Summary

  • Selenium and Playwright drive a real browser from Python: start a browser, navigate, locate elements, act, wait, read results, and always close the browser.
  • Prefer stable locators (test attributes, roles and labels, IDs) over position-based CSS or copied XPath; know the difference between present and visible.
  • Never rely on fixed sleeps: wait for a condition with a timeout (WebDriverWait in Selenium, automatic waiting and expect in Playwright).
  • Handle iframes, popups and dialogs explicitly, and look for a JSON endpoint before automating a scraper.
  • Use the page object pattern to keep selectors in one place and tests readable; combine with pytest fixtures.
  • Automate your own sites and respect other sites' rules; capture screenshots and traces when something fails.

Next: Natural Language Processing with NLTK and spaCy, where you turn text into data a program can analyze.