Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 39 ยท HTTP and HTML Basics

HTTP Requests and Responses

Learn how HTTP works for web scrapers: methods, request and response anatomy, headers, bodies and how a browser page load turns into many requests.

Every time your scraper "visits" a page, it speaks HTTP, the HyperText Transfer Protocol. A client (your script or a browser) sends a request; a server sends back a response. Once you understand the shape of these two messages, libraries such as requests stop being magic, and debugging blocked or empty responses becomes much easier.

This lesson walks through a request and a response line by line, then shows how to build and read them in Python without any network, using the standard library's parsing tools.

The request-response cycle

HTTP is a text-based conversation over a connection (usually encrypted with TLS, which is what the "S" in HTTPS means). The client always speaks first.

sequenceDiagram
    participant C as Your scraper
    participant S as Web server
    C->>S: GET /books?page=2 HTTP/1.1<br/>Host: example.com<br/>User-Agent: ...
    S-->>C: HTTP/1.1 200 OK<br/>Content-Type: text/html<br/><br/>(HTML body)

One request, one response. The connection may be reused for the next request.

Anatomy of a request

A request has four parts:

  1. Request line: the method, the path (with query string) and the protocol version.
  2. Headers: Name: value lines giving extra information.
  3. A blank line.
  4. An optional body, used by methods like POST.
TEXT
GET /books?page=2 HTTP/1.1
Host: example.com
User-Agent: price-tracker-demo/1.0
Accept: text/html
Accept-Language: en-US,en;q=0.9

The Host header names the site (one server can host many). Accept says which content types the client can handle. Accept-Language hints at the preferred language, and some sites return different content based on it.

HTTP methods

The method tells the server what you want to do. The ones you will meet:

Method Meaning Body? Typical scraping use
GET Read a resource No Fetching pages and JSON
POST Send data to be processed Yes Submitting search forms, login, GraphQL
HEAD Like GET but headers only No Checking if a page changed or exists
PUT Replace a resource Yes Rare for scraping
DELETE Remove a resource Maybe Never for scraping
OPTIONS Ask what is allowed No Seen in browser traffic

GET requests should be safe: they must not change anything on the server. That is why scrapers use them for almost everything and why a crawler that follows links only makes GET requests.

Anatomy of a response

A response mirrors the request:

TEXT
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 5123
Cache-Control: max-age=600
Set-Cookie: session=abc123; HttpOnly

<!doctype html>
<html> ... </html>
  1. Status line: protocol, a three-digit status code and a reason phrase.
  2. Headers: metadata about the response.
  3. Blank line, then the body: the HTML, JSON, image or file itself.

The next lesson covers status codes, headers and cookies in depth. For now remember that the body is what you parse, and the headers tell you how to interpret it (for example, the character encoding in Content-Type).

Reading a raw response in Python

The email.parser module in the standard library understands the same header format HTTP uses, so we can split a raw response text into its parts. This is exactly what a client library does for you internally.

Python Take a raw response apart

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A real client does the same split: status line, headers as a case-insensitive mapping, then the body. Header names are case-insensitive (content-type equals Content-Type), which is why requests gives you a case-insensitive dictionary.

Building a request with the standard library

Python's urllib.request.Request object lets you build a request, headers and all, without sending it. That is handy for understanding what a library would transmit:

Python Build a request without sending it

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

To actually send it you would call urllib.request.urlopen(req), or with requests:

Python
import requests

r = requests.get(
    "https://example.com/books",
    params={"page": 2},
    headers={"User-Agent": "price-tracker-demo/1.0"},
    timeout=10,
)
print(r.status_code, r.headers["Content-Type"])
print(r.text[:200])

What one page load really involves

When you type a URL into a browser, it makes many requests: the HTML first, then every stylesheet, script, image and font the HTML mentions, and then extra fetch calls made by JavaScript. A scraper that only calls requests.get(url) receives just the first one, the HTML document. That explains a common surprise: "I can see the prices in my browser but my scraper cannot." The prices came from one of the later requests. Open the Network panel of your browser's developer tools to see the whole list, and look for the one that carries your data; chapter 5 shows how to call it directly.

Persistent connections and sessions

HTTP/1.1 keeps the connection open for several requests, which saves time on the handshake. requests.Session() gives you the same benefit and also remembers cookies and default headers between calls:

Python
import requests

session = requests.Session()
session.headers.update({"User-Agent": "price-tracker-demo/1.0"})

for page in range(1, 4):
    r = session.get("https://example.com/books", params={"page": page}, timeout=10)
    r.raise_for_status()
    print(page, len(r.text))

Use a session whenever you request more than one page from the same site.

HTTP versions

You will see HTTP/1.1, HTTP/2 and HTTP/3. They differ in how bytes travel over the wire (multiplexing many requests over one connection, binary framing, QUIC), but they all carry the same ideas: methods, status codes, headers and bodies. Most scraping libraries hide the version. httpx can opt into HTTP/2 if you install its extra.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which part of a request names the site being contacted when one server hosts many sites?

    The Host header tells the server which website the request is for.
  2. 2. Which HTTP method should a scraper use to read a normal page?

    GET is for reading and is defined as safe: it should not change anything on the server.
  3. 3. You see prices in your browser but not in response.text. Why might this happen?

    A page load is many requests. Your scraper received only the first document, and the data may come from a separate JSON request.
  4. 4. Why use requests.Session for many pages from one site?

    A session reuses the connection (faster) and carries cookies and headers from one call to the next.

Summary

  • HTTP is a request and response conversation; the client always starts.
  • A request is a request line, headers and an optional body; a response is a status line, headers and a body.
  • Scrapers mostly use GET; POST appears for forms and APIs.
  • Header names are case-insensitive, and Content-Type tells you the encoding.
  • A browser page load is many requests; your scraper may need a later one to get the data.
  • Use requests.Session for repeated requests to one site.

Next: Status Codes, Headers and Cookies