Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 39 ยท Dynamic Pages and Browser Automation

Selenium Basics

Control a real browser from Python with Selenium: start headless Chrome, locate elements, read text, click and fill forms, then hand the HTML to BeautifulSoup.

Selenium drives a real web browser from your code. It opens Chrome or Firefox, loads the page, runs all its JavaScript, and lets you click, type, scroll and read the result. Because it is a real browser, it sees exactly what a person sees, including content that only appears after scripts run.

It was built for testing web applications, and it is a heavy tool for scraping: each page takes seconds and the browser eats memory. Use it when the cheaper options from JavaScript-rendered pages fail. Selenium needs a real browser, so the code blocks that start one are read-only here. You can still run the parts that do not need a browser: the XPath and CSS locators we use are the same ones you test with lxml and BeautifulSoup, and the waiting logic can be practiced with a stand-in.

Setup

Terminal
pip install selenium

Selenium 4.6 and newer includes Selenium Manager, which downloads a matching driver (such as chromedriver) automatically. You only need Chrome or Firefox installed.

Your first script

Read-only: run this locally.

Python
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By

options = Options()
options.add_argument("--headless=new")        # no visible window
options.add_argument("--window-size=1280,900")
options.add_argument("--user-agent=DemoBot/1.0 (+https://example.org/bot)")

driver = webdriver.Chrome(options=options)
try:
    driver.get("https://example.com/")
    print(driver.title)
    heading = driver.find_element(By.CSS_SELECTOR, "h1")
    print(heading.text)
finally:
    driver.quit()                              # always close the browser

The try/finally matters. A crashed script that never calls quit() leaves browser processes running in the background and eats your RAM. A context manager makes this tidier: with webdriver.Chrome(options=options) as driver: quits for you in Selenium 4.

Finding elements

find_element returns the first match (or raises NoSuchElementException); find_elements returns a list (possibly empty).

Locator Example
By.CSS_SELECTOR driver.find_elements(By.CSS_SELECTOR, "li.product > a")
By.XPATH driver.find_element(By.XPATH, "//button[contains(., 'Load more')]")
By.ID driver.find_element(By.ID, "search")
By.NAME driver.find_element(By.NAME, "q")
By.LINK_TEXT driver.find_element(By.LINK_TEXT, "Next")
By.TAG_NAME driver.find_elements(By.TAG_NAME, "a")

Prefer CSS selectors for readability and XPath when you need text matching or to walk upward in the tree. You can design and test both without a browser:

Python Test CSS and XPath locators offline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Reading data

Once you hold a WebElement you can read it with:

  • element.text: the visible text (hidden text is excluded, which is often what you want).
  • element.get_attribute("href"): an attribute or property.
  • element.is_displayed(), element.is_enabled().
Python
rows = []
for card in driver.find_elements(By.CSS_SELECTOR, "li.product"):
    rows.append({
        "name": card.find_element(By.CSS_SELECTOR, "h2").text,
        "url": card.find_element(By.TAG_NAME, "a").get_attribute("href"),
    })

The best trick: hand the HTML to BeautifulSoup

Selenium's element API makes one round trip to the browser for every call, which is slow across hundreds of elements. Better: let the browser render the page, grab the whole HTML once, and parse it with BeautifulSoup like any other page.

Python
html = driver.page_source
soup = BeautifulSoup(html, "lxml")
titles = [h.get_text(strip=True) for h in soup.select("li.product h2")]

This keeps your parsing code identical to everything in Parsing with BeautifulSoup, testable on saved files, and uses the browser only for what only a browser can do: rendering.

Interacting

Python
from selenium.webdriver.common.keys import Keys

box = driver.find_element(By.NAME, "q")
box.clear()
box.send_keys("trail shoes", Keys.ENTER)       # type and press Enter

driver.find_element(By.CSS_SELECTOR, "button.load-more").click()
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

execute_script runs JavaScript in the page and can return values, which is handy for scrolling and for reading things that are awkward to locate.

Wait for the page, never sleep blindly

The most common Selenium bug is acting on an element before it exists. time.sleep(5) is both slow (waits even if the page is ready) and unreliable (not enough on a slow day). Use an explicit wait: poll for a condition up to a timeout.

Python
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, timeout=10)
first = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "li.product")))
button = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.load-more")))
button.click()

The idea behind WebDriverWait is a simple polling loop. Here it is with a simulated page that "finishes loading" after a moment, so you can see how it works and what the timeout error looks like:

Python How an explicit wait works

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Selenium's version raises TimeoutException with the same logic, and the expected_conditions helpers are ready-made condition functions. More patterns in Waits, scrolling and interactions.

Reducing the cost

  • Block images and fonts (Chrome prefs) if you only need text.
  • Reuse one driver for many pages instead of starting a browser per page.
  • Check page source and the Network tab for an API first; a browser is rarely needed for every page of a site.
  • Keep headless runs identifiable: set an honest User-Agent.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why call driver.quit() in a finally block?

    Without quit(), crashed scripts leave browser processes running.
  2. 2. Why is time.sleep(5) a poor way to wait for content?

    Explicit waits poll for a condition and continue the moment it is true, up to a timeout.
  3. 3. How can you make parsing faster and more testable when using Selenium?

    One page_source call avoids many browser round trips and reuses your normal parsing code.
  4. 4. What does find_elements return when nothing matches?

    find_element raises an exception for no match, while find_elements returns an empty list.

Summary

  • Selenium drives a real browser, so JavaScript-built content is visible, at the cost of speed and memory.
  • Start the driver headless, wrap it in try/finally (or a with block) and always quit.
  • Locate with CSS or XPath; use WebDriverWait with expected_conditions instead of sleep.
  • Fetch driver.page_source and parse it with BeautifulSoup.
  • Use Selenium only when cheaper approaches fail.

Next: Playwright Basics