Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 39 ยท Dynamic Pages and Browser Automation

Waits, Scrolling and Interactions

Handle infinite scroll, load-more buttons, popups and flaky elements in browser automation, with the stop rules and retry patterns that keep scrapers reliable.

Dynamic pages do not hand you everything at once. A list grows as you scroll, a button reveals the next batch, a cookie banner covers the thing you want to click, and an element you grabbed a moment ago vanishes when the page redraws. Handling these situations is what separates a script that works once from one that works every night.

This lesson collects the interaction patterns you need with Selenium or Playwright: infinite scroll, "load more", waiting for the right thing, closing overlays and recovering from flaky steps. The browser calls are shown as read-only code, and each pattern comes with a runnable simulation of the logic so you can see the stop conditions working.

Infinite scroll

The scroll loop is always the same: scroll down, wait for new content, check whether anything new arrived, repeat. The crucial part is the stop rule. Compare either the page height or the count of items before and after each scroll, and stop when it no longer grows, with a hard ceiling as a safety net.

Playwright version (read-only):

Python
def scroll_to_end(page, item_selector, max_rounds=40):
    previous = 0
    for _ in range(max_rounds):
        page.mouse.wheel(0, 4000)
        page.wait_for_timeout(800)                # give lazy loading a moment
        count = page.locator(item_selector).count()
        if count == previous:
            break                                  # nothing new: we are at the end
        previous = count
    return previous

Selenium version of the scroll itself:

Python
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
height = driver.execute_script("return document.body.scrollHeight")

The logic can be tested against a fake page that loads ten items per scroll up to a total of 47:

Python Scroll until nothing new appears

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

patience handles slow networks: a single quiet round might just mean the next batch is late. Raise it for slow sites.

"Load more" buttons

Same idea, with a click instead of a scroll. Stop when the button disappears or becomes disabled:

Python
while True:
    button = page.get_by_role("button", name="Load more")
    if button.count() == 0 or not button.is_enabled():
        break
    button.click()
    page.wait_for_load_state("networkidle")

If the data comes from a paginated JSON endpoint behind that button, it is usually better to call the endpoint directly (Pagination patterns).

Wait for the right thing

Choose the weakest wait that is still correct. From worst to best:

  1. A fixed sleep: slow and brittle. Avoid.
  2. networkidle (no requests for a while): convenient but unreliable on pages with polling, analytics or websockets that never go quiet.
  3. Wait for a specific element or text that signals "the data is here": the most reliable.
  4. Wait for a specific response: best when you know the API call.
Python
page.wait_for_selector("li.product", state="visible", timeout=15000)
page.wait_for_function("document.querySelectorAll('li.product').length >= 20")

In Selenium the equivalents are EC.visibility_of_element_located, EC.text_to_be_present_in_element and WebDriverWait(...).until(lambda d: len(d.find_elements(...)) >= 20).

Overlays and popups

Cookie banners, newsletter modals and "allow notifications" prompts block clicks, so your click fails with "element is not clickable at point". Close them first, and only if they are there:

Python
banner = page.get_by_role("button", name="Reject all")
if banner.count():
    banner.first.click()

Choose the privacy-friendly option when you must interact with consent banners, and never try to hide the fact that you are automated.

Flaky steps and retries

Animations, redraws and slow servers make single steps fail occasionally. The robust answer is a small retry wrapper that re-runs the step a few times with a pause, and then gives up with a clear error. Also remember that in Selenium an element reference goes stale when the page redraws: re-find it instead of holding on to it.

Python A retry helper for flaky steps

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The same helper works for any step. Make the exception list narrow: retrying a programming error (like a typo in a selector name) only hides it.

Putting it together: a render-then-parse loop

A reliable browser scraper has these stages, each one testable on its own:

flowchart LR
  A["Open page"] --> B["Dismiss overlays"]
  B --> C["Scroll or click until done"]
  C --> D["Take page HTML"]
  D --> E["Parse with BeautifulSoup"]
  E --> F["Save rows"]

Only the first three stages need the browser. Everything after them is ordinary parsing that you can run on a saved file:

Python Parse the final HTML after scrolling

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the uniqueness check: infinite-scroll pages sometimes render the same item twice, so always verify the final data.

Save a screenshot when things go wrong

When a run fails in the middle of the night, a screenshot and the HTML of the page at that moment are worth more than a stack trace:

Python
try:
    run_scraper(page)
except Exception:
    page.screenshot(path="failure.png", full_page=True)
    open("failure.html", "w", encoding="utf-8").write(page.content())
    raise

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the best stop rule for an infinite-scroll loop?

    Compare the item count (or page height) between rounds and add a ceiling as a safety net.
  2. 2. Why is waiting for a specific element better than waiting for networkidle?

    Waiting for the thing you actually need is both more precise and more reliable.
  3. 3. What should you do when a cookie banner blocks a click?

    Check for the overlay and dismiss it before interacting with elements underneath.
  4. 4. Why should a retry helper catch only specific exceptions?

    Retry transient problems like timeouts; let real bugs fail loudly.

Summary

  • Infinite scroll and load-more loops need a stop rule (no new items) plus a hard limit.
  • Wait for the specific element, text or response you need, not for a fixed time.
  • Dismiss overlays before interacting; choose privacy-friendly options.
  • Wrap flaky steps in a narrow retry helper and re-find elements instead of keeping stale references.
  • Use the browser only to render; parse the resulting HTML with BeautifulSoup.
  • Save a screenshot and the HTML when a run fails.

Next: robots.txt and Sitemaps