Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 35 ยท Projects

Project: Extracting Table Data

Build a table extractor: read rows and cells, walk through pagination safely, clean the values and export CSV, with runnable pandas-free stand-ins.

Tables are everywhere in real applications: order lists, user admin screens, reports, price lists. Sooner or later a test needs to answer questions such as "does the table contain exactly 8 rows?", "is the total in the footer correct?" or "is the list sorted by price?" This project builds a small table extractor that reads every row across several pages, cleans the values and exports them as CSV, using nothing but the Python standard library.

Selenium needs a real browser and driver, so it cannot run on this site. The Selenium code below is read-only. The runnable examples are honest stand-ins: they parse HTML text that looks like what a browser would hand your script, walk through fake pages, and clean and export the data with the same code you would use after a real scrape. Everything after "collect the text" is real and reusable.

Before you start, you should know locators, CSS selectors and explicit waits. If you also want to read the csv module in more depth, that lesson is a good companion.

Four steps to a clean dataset

Read Collect the text of every row and cell.
Page Click Next until there is no Next.
Clean Turn "$1,299.00" into a number.
Export Write one tidy CSV file.

Keep the steps in separate functions so each one can be tested alone.

1 / 2

Step 1: reading rows and cells

A well-formed HTML table has a thead with th cells and a tbody with one tr per record. With Selenium you ask the driver for every row, then ask each row for its cells. Searching inside a row element keeps the lookup relative to that row.

Python
from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get("https://shop.example.test/admin/inventory")

headers = [th.text for th in driver.find_elements(By.CSS_SELECTOR, "table#inventory thead th")]
rows = []
for tr in driver.find_elements(By.CSS_SELECTOR, "table#inventory tbody tr"):
    cells = [td.text for td in tr.find_elements(By.TAG_NAME, "td")]
    rows.append(dict(zip(headers, cells)))

Two details matter:

  • element.text returns only the visible text. A cell hidden with CSS gives an empty string. If you need hidden text, read element.get_attribute("textContent").
  • Each find_elements call is a round trip to the browser. A table with 500 rows and 6 columns costs 3,000 calls. For large tables, run one script and receive plain arrays: driver.execute_script("return [...document.querySelectorAll('#inventory tbody tr')].map(tr => [...tr.cells].map(td => td.textContent.trim()))").

Our stand-in replaces the browser with a small parser. Python's built-in html.parser reads the HTML and records rows, cells and a "next page" link:

Python Stand-in: read a table from HTML text

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The parser is the stand-in for find_elements; the result, a list of header names and a list of row lists, is exactly what the Selenium loop above produces. Notice the stock of the notebook is a dash, not a number. Real tables are full of such surprises, so the cleaning step cannot be skipped.

Step 2: pagination

Large tables are split into pages. The pattern is always the same: collect the rows on the current page, look for an enabled Next control, click it, wait until the new page has really arrived, and repeat.

flowchart TD
  A["Read rows on this page"] --> B{"Enabled Next link?"}
  B -- "yes" --> C["Click Next"]
  C --> D["Wait until the old first row is stale"]
  D --> A
  B -- "no" --> E["Done"]

The loop ends when there is no enabled Next control, not after a fixed number of pages.

The waiting step is the one beginners forget. Right after the click, the old rows are still in the DOM for a moment, so reading immediately returns the previous page again. Holding on to the old first row and waiting for it to go stale is a reliable signal that the table was replaced:

Python
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

all_rows = []
while True:
    all_rows.extend(read_rows(driver))
    next_links = driver.find_elements(By.CSS_SELECTOR, "a.next:not(.disabled)")
    if not next_links:
        break
    old_first_row = driver.find_element(By.CSS_SELECTOR, "#inventory tbody tr")
    next_links[0].click()
    WebDriverWait(driver, 10).until(EC.staleness_of(old_first_row))

This only works when the page replaces the table's rows. If the site updates cells in place, wait for the text of the first cell or a "page 2" indicator to change instead (see Implicit, Explicit and Fluent Waits). Also add a safety cap such as max_pages = 200, so a broken Next link cannot loop forever.

Our stand-in fakes a three-page site with a dictionary of pages and a fetch function that plays the role of "click Next and wait":

Python Stand-in: walk through three pages

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 3: cleaning the values

Everything Selenium gives you is a string. Before you compare, sum or sort, convert each column to the right type and decide what the odd values mean. A clear rule is better than a silent guess: here an em dash in the stock column means "unknown", which becomes None, not zero.

Step 4: exporting CSV and checking the data

The standard csv module handles quoting for you (names with commas, quotes or line breaks). Writing to an in-memory buffer lets you test the export without touching the disk; in your real project you would pass a file opened with newline="" instead. Then you can reload the CSV, which doubles as a check that the export is valid, and answer questions without pandas using csv.DictReader, collections and sorted.

Python Clean, export and analyse without pandas

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the empty stock field in the CSV: None was written as an empty string, and reading it back gives "". Decide early how your data represents "no value" and keep it consistent between writing and reading.

Putting it together

Your real scraper is now a handful of small functions: read_rows(driver) returns the strings, go_to_next_page(driver) returns False at the end, clean(row) converts types, and write_csv(records, file) exports. The test then calls them and asserts, for example, that the row count equals the "Showing N results" label, that no price is negative, and that sorting by a column really orders the values.

A few habits keep this reliable:

  • Assert on counts first. A scraper that silently returns half the rows is worse than one that fails.
  • Prefer test ids on tables. A selector such as [data-testid="inventory"] tbody tr survives redesigns far better than div > div > table:nth-child(3).
  • Use the page's own export or API when it exists. If a site offers a CSV download or a JSON endpoint, a browser is the slow way to get the data.
  • Respect the rules. Reading your own application's tables in a test is normal. Scraping someone else's site may break their terms or load their servers; read Security, Ethics and Scraping Limits first.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does a pagination loop wait for the old first row to go stale after clicking Next?

    Right after the click the old rows can still be in the DOM. Waiting for the staleness of one of them is a signal that the new page has arrived.
  2. 2. What does element.text return for a cell hidden with CSS?

    text returns only what a user could see. textContent gives the raw text of the node, visible or not.
  3. 3. A table has 500 rows and 6 columns. Why might you read it with one execute_script call?

    One script that returns plain arrays replaces thousands of separate requests.
  4. 4. Why use csv.DictWriter or csv.writer instead of joining values with commas yourself?

    A name such as 'Rug, wool' would break a hand-made CSV line. The csv module adds the quotes.

Summary

  • Read a table by finding the rows, then the cells inside each row; text is visible text only.
  • For big tables, one execute_script returning arrays is much faster than many element calls.
  • Loop through pages until there is no enabled Next control, wait for the table to change after each click, and cap the number of pages.
  • Everything comes back as strings: clean types and decide what "no value" means.
  • Export with the csv module and reload the file to check it; assert on row counts first.
  • Only automate sites you own or have permission to test.

Next, bring the pieces of the whole course together in Project: A Small Test Framework.