Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 36 ยท Ingestion and Batch Processing

Ingesting from APIs

Pull data from REST APIs safely: pagination, rate limits, retries with backoff, incremental cursors and storing raw responses, using a simulated API in Python.

Software-as-a-service tools, payment providers and partner systems expose their data through APIs. Pulling from an API is different from reading a file: the data arrives in pages, calls can fail or be throttled, and the server controls how fast you may go. This lesson covers the patterns that make an API ingestor reliable.

Browser sandboxes cannot call real third-party APIs from these examples, so a small Python class below simulates a paginated, occasionally failing API. The ingestion logic you write against it is the same logic you would write against a real HTTP client such as requests or httpx.

A simulated API

The fake API returns records in pages of three, with a cursor pointing at the next page. It also fails on the second call to mimic a temporary server error, and it records a request count.

Python The simulated API

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Pagination

APIs rarely return everything at once. Three common schemes:

Scheme How it works Notes
Offset/limit ?offset=100&limit=50 Simple, but rows inserted mid-run shift pages
Page number ?page=3 Same weakness as offset
Cursor Response carries an opaque next token Most robust; follow until it is empty

Always loop until the API says there is no more, never "until a page is short" or "for N pages".

Retries with exponential backoff

Transient errors (network blips, 503, 429 "too many requests") should be retried, but politely: wait longer after each failure so you do not hammer a struggling server. Do not retry permanent errors such as 400 or 404; they will never succeed.

Python Retry with backoff and a cap

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The code prints the delay instead of sleeping so it finishes instantly here. In production add jitter (a small random amount) so many workers do not retry in lockstep, and respect a Retry-After header when the server sends one.

Rate limits

Providers cap how many calls you can make per minute. Build a limiter into the client rather than hoping you stay under it. A simple version spaces calls by a minimum interval:

Python A minimal rate limiter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real limiters (token buckets) allow short bursts, but the principle is the same: the client slows itself down.

Incremental extraction with a cursor

Pulling the whole history daily is wasteful. Ask only for what changed since last time, using a high-water mark such as updated_since, and save the newest value you saw. The details of incremental loading are in the next lesson.

Python High-water mark state

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Saving the mark after a successful load means a crash causes re-fetching (safe, with idempotent loads) rather than skipping data (unsafe).

Store the raw response

Keep the exact JSON the API returned, with the request parameters and a timestamp, before parsing. When the provider changes a field name or you discover a parsing bug, you can reprocess from the saved responses without calling the API again (and without worrying that the API no longer returns old data).

flowchart LR
    A["Call API"] --> B["Save raw JSON<br/>+ params + time"]
    B --> C["Parse and flatten"]
    C --> D["Load table"]
    A -. "retry on 5xx/429" .-> A

Retries wrap the call; raw storage separates fetching from parsing.

Secrets and etiquette

  • Keep API keys out of code; read them from environment variables or a secret manager.
  • Identify your client with a User-Agent when allowed.
  • Read the provider's terms for limits and permitted use.
  • Log request counts and status codes, never the secret.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which pagination method is most robust against rows being added during extraction?

    A cursor points to a stable position, whereas offsets shift when rows are inserted.
  2. 2. Which errors should you normally retry?

    Transient errors may succeed later. Permanent client errors will not.
  3. 3. Why add jitter to backoff delays?

    Random variation spreads retries out so they do not overload the server together.
  4. 4. When should you save the high-water mark?

    Saving after success means a crash causes a safe re-fetch instead of silently skipping data.

Summary

  • Follow cursors until the API says it is done; do not guess page counts.
  • Retry transient errors with exponential backoff and jitter, and respect rate limits.
  • Extract incrementally using a high-water mark saved only after a successful load.
  • Store raw responses, keep secrets out of code, and log counts and statuses.

Next: Incremental Loading and Change Data Capture