Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 39 ยท Selectors and Data Extraction

Extracting Tables and Lists

Turn HTML tables, nested lists and definition lists into Python dictionaries and CSV: headers, colspan and rowspan, multiple tables, and pandas.read_html.

Tables are the most scraper-friendly structure on the web: rows and columns already look like a dataset. Lists (ul, ol, dl) are almost as easy. This lesson shows how to convert them into Python lists of dictionaries, how to cope with the tricky bits (header rows, merged cells, several tables on a page), and when pandas.read_html does the whole job for you.

Anatomy of an HTML table

HTML
<table id="prices">
  <caption>Weekly prices</caption>
  <thead>
    <tr><th>Item</th><th>Price</th><th>Stock</th></tr>
  </thead>
  <tbody>
    <tr><td>Lamp</td><td>$24.50</td><td>12</td></tr>
    <tr><td>Stand</td><td>$11.00</td><td>0</td></tr>
  </tbody>
  <tfoot><tr><td colspan="2">Total</td><td>12</td></tr></tfoot>
</table>

tr is a row, th a header cell, td a data cell. thead, tbody and tfoot group rows, but many real tables omit them, so write code that works either way.

A generic table-to-records function

The recipe: read the header cells for column names, then zip each data row with them.

Python Table to list of dictionaries

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two design points. First, find_all("tr") searches the whole table, including thead, tbody and tfoot, so the function works whether or not those wrappers exist. Second, the len(cells) == len(header) check silently skips short rows (such as a footer with colspan); it is better to log what you skip, as the cleaning lesson recommends.

Header cells in the first column

Some tables put row labels in th cells on the left (a "stub" column):

Python Row headers and column headers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When a table is a grid with both row and column headings, a nested dictionary (or a pandas DataFrame) fits better than a list of records.

colspan and rowspan

Merged cells break the "one cell per column" assumption. colspan="2" means the cell covers two columns; rowspan="2" means it covers two rows, so the next row has fewer cells. A robust approach is to expand the table into a full rectangular grid:

Python Expand colspan and rowspan

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

After expansion every row has the same number of cells, so you can read headers from the first one or two rows and zip them normally. This function is more advanced than most scrapers need; you only reach for it when a table uses merged cells.

Several tables on one page

Pages often contain many tables (layout tables, data tables, tables of contents). Pick the right one by an identifier, by its position, or by its caption or header text:

Python
soup.find("table", id="prices")
soup.select_one("table.wikitable")
soup.find_all("table")[2]
soup.find("caption", string="Weekly prices").find_parent("table")

Selecting by header text is the sturdiest because it survives layout reordering: loop over soup.find_all("table") and keep the one whose first row contains the column names you expect.

Lists

An unordered or ordered list becomes a Python list in one expression. Nested lists need recursion.

Python Lists, nested lists and definition lists

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For the nested list, li.find_all(string=True, recursive=False) collects only the direct text of the li ("Products") and skips the text inside the nested ul. Without recursive=False you would glue the child labels onto the parent.

Writing a table to CSV

Records fit csv.DictWriter directly (see Your First Scraper). For analysis, load them into pandas:

Python
import pandas as pd

df = pd.DataFrame(records)
df["Price"] = df["Price"].str.lstrip("$").astype(float)
df["Stock"] = df["Stock"].astype(int)
print(df.describe())

pandas.read_html

If you simply want the tables, pandas can parse them for you. pd.read_html takes a URL, a file or an HTML string and returns a list of DataFrames, one per table. It needs lxml (or bs4 with html5lib) installed. In real code you pass the page text wrapped in StringIO:

Python
from io import StringIO
import pandas as pd

tables = pd.read_html(StringIO(html))           # all tables
prices = pd.read_html(StringIO(html), match="Weekly prices")[0]

It handles colspan, rowspan and multi-row headers for you, and the match= argument selects tables containing some text. Its limits: it only reads actual table elements (not card grids), it guesses data types, and it cannot see data that JavaScript adds. Use it for quick work; write your own parser when you need control over odd cells, links inside cells or per-row logic. See the pandas tutorial for what to do with the resulting DataFrame.

Often a cell holds a link or image and you need the URL as well as the text:

Python
for row in table.find_all("tr")[1:]:
    cells = row.find_all("td")
    link = cells[0].find("a")
    record = {
        "name": cells[0].get_text(strip=True),
        "url": link["href"] if link else None,
        "price": cells[1].get_text(strip=True),
    }

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why use table.find_all('tr') rather than table.tbody.find_all('tr')?

    thead, tbody and tfoot are optional in source HTML, so searching the whole table for tr works in both cases.
  2. 2. A cell has rowspan='2'. What happens to the next row?

    The merged cell occupies a slot in the next row, so that row lists fewer td elements.
  3. 3. What does pandas.read_html return?

    read_html returns a list because a page can contain several tables.
  4. 4. Why use find_all(string=True, recursive=False) for a nested list item's own label?

    recursive=False limits the search to direct children, so text within nested ul elements is not included.

Summary

  • Convert tables with a header row plus zip(header, cells); search tr across the whole table so optional tbody does not matter.
  • Handle row headings, colspan and rowspan by expanding to a rectangular grid when needed.
  • Choose the right table by id, caption or header text rather than position alone.
  • Lists become Python lists; nested lists need recursion and recursive=False; dl pairs dt with dd.
  • pandas.read_html quickly returns every table as a DataFrame, but cannot see JavaScript-built data.

Next: Extracting Links, Images and Metadata