Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 39 ยท Requests and APIs

Pagination Patterns

Scrape every page of a list: page numbers, offsets, cursors, next links, Link headers and infinite scroll. Reusable Python generators with offline demos.

Almost every interesting dataset is split across pages. A shop shows 24 products at a time, an API returns 100 records per call, a forum thread spreads over twenty screens. To collect everything you need a loop that knows how to ask for "the next chunk" and, just as important, when to stop.

There are only a handful of pagination styles. Once you can recognize them, any site becomes a variation on a pattern you already have. In this lesson we write a small generator for each style and run it against a fake in-memory server, so everything works offline.

The five common styles

Style URL or request looks like How you stop
Page number ?page=3 Empty list, or page beyond total_pages
Offset and limit ?offset=40&limit=20 Fewer items than limit
Cursor or token ?cursor=eyJpZCI6NDB9 Response has no next_cursor
Next link <a rel="next" href="..."> or JSON "next": url Link missing
Infinite scroll Extra requests fired while scrolling Same as one of the above, found in DevTools

Infinite scroll is not a separate technique. The page calls a JSON endpoint with a page number or cursor each time you scroll, so you find that endpoint in the Network tab (JSON APIs) and use one of the first four patterns.

A fake server to practice on

Our stand-in api(...) returns data the way a real endpoint would. Every example below calls it instead of requests.get, so swapping in the real thing later means changing a single line.

Page numbers

Python Page-number pagination as a generator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A generator is ideal here: the caller can process records as they arrive, stop early, or write each one to disk without holding everything in memory. Notice the max_pages ceiling. Every pagination loop needs a safety limit, because a site that ignores out-of-range pages and keeps returning the last one would trap you forever.

Offset and limit

Offset pagination is the same idea counted in records instead of pages. The stop condition is a short page:

Python Offset and limit

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

If new records are added while you scrape, offsets shift and you may see duplicates or miss items. Deduplicate by ID afterwards (Deduplicating and updating).

Cursors

A cursor is an opaque bookmark. You never compute it; you copy whatever the server sent and send it back. It is the most robust style for fast-changing data.

Python Cursor pagination

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The scraper treats the cursor as a black box: it only checks whether one came back. Real cursors are often base64 or encrypted strings that you must not try to forge.

HTML sites usually mark the next page with a link, often rel="next" or a button labeled "Next". Following links rather than counting is the most faithful way to crawl, because it does exactly what a human would do. Make sure to turn the relative href into an absolute URL with urljoin:

Python Follow the next link

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

APIs sometimes put the next URL in an HTTP Link header instead of the body. requests parses it for you into response.links:

Python
resp = requests.get("https://api.example.com/items", timeout=10)
next_url = resp.links.get("next", {}).get("url")

You can try the parser on a header string:

Python Parse a Link header

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Knowing when to stop

Combine these signals instead of trusting just one:

  • An empty list, a short page, a missing cursor or a missing next link.
  • A total or total_pages field, if the response has one.
  • A hard max_pages limit you choose.
  • A "no new IDs on this page" check, which catches sites that repeat the last page.
flowchart TD
  A["Request page"] --> B{"Items returned?"}
  B -- "no" --> Z["Stop"]
  B -- "yes" --> C{"Any unseen IDs?"}
  C -- "no" --> Z
  C -- "yes" --> D["Save items"]
  D --> E{"Page limit reached?"}
  E -- "yes" --> Z
  E -- "no" --> A

A robust loop has more than one way out.

Be gentle while paging

Fifty pages fetched in one second looks like an attack. Pause between requests, and save progress (the last page or cursor) so a crash does not force you to start over. We build both in Rate limiting and retries and Building a crawler.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which pagination style gives you an opaque token that you send back unchanged?

    A cursor is an opaque bookmark issued by the server; you pass it back without interpreting it.
  2. 2. Why should a page-number loop have a maximum page count?

    A hard ceiling protects you from loops caused by sites that ignore out-of-range pages.
  3. 3. How do you turn a relative href like /list?p=3 into a full URL?

    urljoin resolves relative links against the current page URL correctly, including paths and query-only links.
  4. 4. With offset pagination, how can you tell you reached the last page?

    A page shorter than the requested limit means there is nothing more to fetch.

Summary

  • Pagination comes in a few styles: page number, offset, cursor, next link, and infinite scroll (which is one of the others in disguise).
  • Write each as a generator so callers can stream, stop early or save as they go.
  • Always add a stop rule beyond "the server says so": a page ceiling and a seen-IDs or seen-URLs check.
  • Use urljoin for relative links and response.links for Link headers.
  • Pause between requests and remember your position so you can resume.

Next: Sessions, Cookies and Login