Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 39 ยท Requests and APIs

The requests Library

Fetch pages with Python requests: GET and params, headers, timeouts, status codes, raise_for_status, sessions and retries. Includes offline examples you can run.

Until now you have parsed HTML that was already in a string. Real scraping starts one step earlier: your program has to ask a server for the page. In Python the standard tool for that is requests, a small library that turns an HTTP conversation into a few readable lines of code.

This lesson covers the parts of requests that matter for scraping: building a request, reading the response, handling errors, and keeping a session. The examples on this page run without a network connection. We build requests and responses by hand so you can see exactly what the library does. The code that really talks to a server is shown in read-only blocks, because the in-browser runner cannot open connections.

Installing and the first request

requests is a third-party package. Install it once in your project's virtual environment:

Terminal
pip install requests beautifulsoup4 lxml

A minimal fetch looks like this (read-only, run it on your own machine):

Python
import requests

response = requests.get("https://example.com/", timeout=10)
print(response.status_code)       # 200
print(response.headers["Content-Type"])
html = response.text              # decoded text

Three things to notice. requests.get returns a Response object, not a string. The status code tells you how the server felt about your request. And timeout=10 is there on purpose: without a timeout, requests waits forever, and a single stuck connection can freeze a whole scraper.

What is inside a Response

Attribute What it gives you
status_code Integer such as 200, 404, 429, 503
ok True when the status is below 400
text The body decoded to str using the detected encoding
content The raw body as bytes (images, PDFs, or when you pick the encoding yourself)
json() Parses the body as JSON (see JSON APIs)
headers Case-insensitive dictionary of response headers
url The final URL after redirects
history The redirect responses that led here
cookies Cookies the server set
elapsed How long the request took

The next example builds a Response by hand, which is something you would only do in tests, but it lets us explore raise_for_status() without a network.

Python Responses and raise_for_status

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

raise_for_status() turns 4xx and 5xx answers into a requests.HTTPError. It is the cheapest way to make sure you never parse an error page as if it were real data.

Query parameters

Never glue query strings together by hand. Pass a dictionary and requests encodes it correctly (spaces, ampersands, non-ASCII characters):

Python
requests.get("https://shop.test/search", params={"q": "red shoes", "page": 2})

You can check the exact URL without sending anything by preparing the request:

Python Preview a request without sending it

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A list value becomes a repeated parameter (sort=price&sort=name), and the special characters in red shoes & socks are escaped. Previewing prepared requests is also a good debugging trick when a server refuses you and you want to compare your request with the one your browser sends.

Headers

By default requests introduces itself as python-requests/2.x. Many sites treat that as a bot. You should send an honest, descriptive User-Agent that says who you are and how to contact you, plus the headers a normal client sends:

Python
headers = {
    "User-Agent": "PriceWatchBot/1.0 (+https://example.org/bot; [email protected])",
    "Accept": "text/html,application/xhtml+xml",
    "Accept-Language": "en-US,en;q=0.9",
}
response = requests.get(url, headers=headers, timeout=10)

Pretending to be Chrome is a common trick, but it is dishonest and often unnecessary. Legal and ethical scraping explains why an identifiable agent is the better default.

Sessions

A requests.Session keeps cookies, default headers and open connections between calls. It is faster (connection reuse) and it is required for any site that sets a cookie on the first request and expects it on the next.

Python Session defaults are merged into each request

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Session defaults are merged with per-request values, and the per-request value wins. In real code you create one session per site and reuse it for every page.

Timeouts, redirects and retries

Pass timeout=(connect, read) for fine control, for example timeout=(3.05, 15). Redirects are followed automatically for GET; response.history lists them and allow_redirects=False turns that off.

For flaky servers, mount a retry policy on the session. This retries only idempotent GET requests, and only on typical transient statuses:

Python
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

retry = Retry(
    total=4,
    backoff_factor=1.0,                  # grows with every attempt
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods=["GET", "HEAD"],
    respect_retry_after_header=True,
)
session = requests.Session()
session.mount("https://", HTTPAdapter(max_retries=retry))
session.mount("http://", HTTPAdapter(max_retries=retry))

respect_retry_after_header honors the server's Retry-After header on 429 Too Many Requests, which is exactly what a polite scraper should do. We build our own backoff logic in Rate limiting and retries.

Encoding and big downloads

response.text guesses the encoding from the headers. If the page looks garbled, set it explicitly before reading .text:

Python
response.encoding = "utf-8"          # or response.apparent_encoding

For large files, stream instead of loading everything into memory:

Python
with requests.get(url, stream=True, timeout=30) as r:
    r.raise_for_status()
    with open("report.pdf", "wb") as f:
        for chunk in r.iter_content(chunk_size=65536):
            f.write(chunk)

A safe fetch helper

Put the good habits in one function and use it everywhere:

Python
def fetch(session, url, **kwargs):
    response = session.get(url, timeout=(3.05, 15), **kwargs)
    response.raise_for_status()
    return response.text

Keep the network layer (fetch) separate from the parsing layer (a function that takes an HTML string and returns data). You can then test parsing against saved HTML files and never hit the live site while developing.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why should every requests call in a scraper have a timeout?

    requests has no default timeout, so a hung server can block your program indefinitely.
  2. 2. What does response.raise_for_status() do?

    It raises requests.HTTPError when the status code indicates a client or server error, so you never parse an error page by accident.
  3. 3. Why pass query parameters with params={...} instead of building the URL string yourself?

    Manual string joining breaks on spaces, ampersands and non-ASCII text; params handles the escaping.
  4. 4. What is the main benefit of using a Session?

    A Session persists cookies, headers and TCP connections across requests.

Summary

  • requests.get(url, timeout=...) returns a Response; always set a timeout.
  • Check status_code or call raise_for_status() before parsing.
  • Use params= for query strings and an honest User-Agent header.
  • A Session shares cookies, headers and connections; mount a Retry adapter for transient failures.
  • Prepared requests let you inspect what will be sent without any network access.
  • Separate fetching from parsing so each can be tested alone.

Next: Finding and Using JSON APIs