Skip to content
Tutorials Center
My learning

Topics

Lesson 10 of 39 ยท Parsing with BeautifulSoup

Navigating the Parse Tree

Walk the BeautifulSoup tree: children, descendants, parent, parents, next and previous siblings, next_element and a practical label-and-value scraping pattern.

Searching by name or class is only half of scraping. Often the thing you want has no useful class of its own, but sits right next to something that does: the number after a label, the paragraph below a heading, the cell next to a header cell. BeautifulSoup lets you move around the tree from any tag you hold, going down to children, up to parents and sideways to siblings.

This lesson uses one running example, a product spec page, to demonstrate each direction.

The tree we will walk

HTML
<section id="specs">
  <h2>Specifications</h2>
  <dl>
    <dt>Weight</dt><dd>1.2 kg</dd>
    <dt>Color</dt><dd>Black</dd>
    <dt>Warranty</dt><dd>2 years</dd>
  </dl>
  <p class="note">Specs may vary.</p>
</section>
flowchart TD
    S["section #specs"] --> H["h2"]
    S --> DL["dl"]
    S --> P["p .note"]
    DL --> DT1["dt Weight"]
    DL --> DD1["dd 1.2 kg"]
    DL --> DT2["dt Color"]
    DL --> DD2["dd Black"]

Downwards: children and descendants. Upwards: parents. Sideways: siblings.

Going down: children and descendants

  • .contents: a list of direct children (including whitespace text nodes).
  • .children: an iterator over the same.
  • .descendants: every node below, at any depth.
  • find_all(recursive=False): only direct child tags, skipping text nodes. This is usually what you want.
Python Children versus descendants

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The raw .contents list includes the newline-and-indent strings between tags, so its length is larger than the number of child tags. This catches beginners out constantly. Prefer find_all(recursive=False) or .find_all(True, recursive=False) for tags only.

Going up: parent and parents

  • .parent: the immediate parent tag.
  • .parents: a generator yielding the parent, grandparent, and so on up to the BeautifulSoup object.
  • find_parent(name, class_=...): the nearest ancestor matching a filter.

Going up is useful when you can locate a distinctive child but need the whole record around it. For instance, find the "Add to cart" button, then climb to the product card:

Python From a button up to its card

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

find_parent("div", class_="card") skipped the intermediate actions div and returned the matching ancestor. The parents list ends with [document], the soup object itself.

Going sideways: siblings

Siblings share a parent.

  • .next_sibling / .previous_sibling: the adjacent node, which is often a whitespace string.
  • .next_siblings / .previous_siblings: generators.
  • find_next_sibling(...) / find_previous_sibling(...): the next sibling tag matching a filter. Use these.
  • find_next_siblings(...): all following sibling tags.

The label-and-value pattern is the canonical use: find the label by its text, then jump to the neighbor that holds the value.

Python Label and value with siblings

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first line shows the trap: because a newline sits between dt and dd, next_sibling returns the newline string, not the tag. find_next_sibling("dd") is immune to that, because it skips text nodes and non-matching tags.

Document order: next_element and previous_element

Siblings follow the tree structure. Elements follow the order in which the parser read the document, going into children first:

  • .next_element, .previous_element
  • .next_elements, .previous_elements
  • find_next(...), find_all_next(...), find_previous(...), find_all_previous(...)

find_next("p") returns the next p anywhere after the current position in the document, regardless of nesting. That helps with flat pages where headings and paragraphs are siblings of each other, as in many articles:

Python Collecting content under each heading

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loop walks the following siblings of each heading and stops at the next h2. This "section until the next heading" pattern is the standard way to group flat content.

Which direction should I use?

You have You want Use
A container Its direct sub-tags find_all(recursive=False)
A container Everything below find_all(...), .descendants
A child Its record find_parent(...)
A label The adjacent value find_next_sibling(...)
A heading The content until the next heading find_next_siblings() with a break
Any tag The next occurrence in document order find_next(...)

Prefer navigating from a stable anchor (an id, a label text) and moving a short, clear distance. Long chains such as tag.parent.parent.next_sibling.next_sibling.div break the moment the layout shifts. When a CSS selector can express the path, use it.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does tag.next_sibling often return a newline string instead of the next tag?

    Indentation and line breaks between tags become NavigableString nodes, which count as siblings.
  2. 2. Which call returns the next sibling TAG named dd, skipping whitespace?

    find_next_sibling searches following siblings for a tag matching the filter and ignores text nodes.
  3. 3. You located an Add to cart button and need the whole product card containing it. Which method fits?

    find_parent climbs the ancestors until it finds one that matches the filter.
  4. 4. What does find_all(recursive=False) do?

    With recursive=False, only the direct child tags of the starting tag are examined.

Summary

  • Go down with .children, .descendants and find_all(recursive=False); remember .contents includes whitespace text nodes.
  • Go up with .parent, .parents and find_parent.
  • Go sideways with find_next_sibling(s) and find_previous_sibling(s), not the raw .next_sibling.
  • Use document-order methods (find_next) for flat pages, and stop at the next heading to group content.
  • Anchor on something stable and keep navigation chains short.

Next: find() and find_all()