Skip to content
Tutorials Center
My learning

Topics

Lesson 6 of 39 · HTTP and HTML Basics

Status Codes, Headers and Cookies

Master the HTTP details a scraper depends on: status code families, redirects, 403 and 429 blocks, the headers that matter, content encoding and how cookies work.

A scraper that only looks at the body of a response will eventually break in confusing ways. A "200 OK" can hide a login page; a "403" means you were recognized as a bot; a missing cookie makes every page redirect to a consent screen. The status code and the headers explain these situations. This lesson shows which values matter, what they mean and how to react to each in code.

Status code families

The first digit of the status code tells you the category.

Family Meaning Examples
1xx Informational 100 Continue
2xx Success 200 OK, 204 No Content
3xx Redirection 301, 302, 304, 307, 308
4xx Client error: you did something wrong 400, 401, 403, 404, 429
5xx Server error: they have a problem 500, 502, 503, 504

The codes you will see most often while scraping:

  • 200 OK: success. Still check that the body is what you expect.
  • 301 / 308 (permanent redirect) and 302 / 307 (temporary redirect): the new address is in the Location header. requests follows redirects by default.
  • 304 Not Modified: the page has not changed since your cached copy (see conditional requests below).
  • 400 Bad Request: the server could not understand your request, often malformed parameters.
  • 401 Unauthorized: authentication is needed.
  • 403 Forbidden: the server understood and refuses. Common when it detects a bot.
  • 404 Not Found: the page does not exist. Normal in crawls; log it and move on.
  • 429 Too Many Requests: you are being rate limited. Slow down.
  • 500 / 502 / 503 / 504: server trouble; retrying later with a delay often works.

How to react to a status

2xx Parse the body.
3xx Follow Location, watch for loops.
4xx Fix the request or stop.
5xx Wait, then retry a few times.
1 / 2

A status decision function

You can turn the table above into a small, testable function. It runs here with made-up codes, no network needed.

Python Decide what to do for each status

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With requests, the matching tools are response.status_code, response.ok (true for anything below 400), and response.raise_for_status(), which raises requests.HTTPError for 4xx and 5xx. A robust fetch wrapper checks the status and decides per category.

Redirects

When a server answers 301 or 302 with Location: /new-path, the client repeats the request there. requests does this for you and records the chain in response.history. Redirects matter to scrapers because:

  • The final URL (response.url) may differ from the one you asked for; use it as the base when resolving relative links.
  • A redirect to a login or consent page means you did not get the content you wanted, even though the final status is 200.
  • Redirect loops can trap a naive crawler; requests stops after 30 redirects.

Headers that matter

Request headers you set:

Header Purpose
User-Agent Names your client. Be honest and include contact details.
Accept Content types you can handle, such as text/html or application/json.
Accept-Language Preferred language, which can change the page.
Accept-Encoding Compression you support (gzip, deflate, br). Libraries set this for you.
Referer The page that linked here; some sites check it.
Cookie Cookies being sent back.
If-None-Match, If-Modified-Since Conditional requests that save bandwidth.

Response headers you read:

Header Meaning
Content-Type Format and charset, for example text/html; charset=utf-8.
Content-Length Size of the body in bytes.
Content-Encoding Compression applied (gzip, br); the library decompresses for you.
Location Target of a redirect.
Retry-After Seconds (or a date) to wait before trying again, sent with 429 and 503.
Set-Cookie Asks you to store a cookie.
ETag, Last-Modified Version markers for caching.
Cache-Control How long the response may be reused.

Conditional requests: skip unchanged pages

If you re-scrape the same page daily, you do not need to download it again when it has not changed. Save the ETag from the first response and send it back as If-None-Match. If nothing changed the server answers 304 Not Modified with no body.

Python
import requests

url = "https://example.com/prices"
r1 = requests.get(url, timeout=10)
etag = r1.headers.get("ETag")

headers = {"If-None-Match": etag} if etag else {}
r2 = requests.get(url, headers=headers, timeout=10)
if r2.status_code == 304:
    print("Unchanged, reuse the saved copy")
else:
    print("Changed, parse the new body")

This is polite (less bandwidth for the site) and fast for you.

Encoding

Content-Type: text/html; charset=utf-8 tells you how the bytes map to characters. If a server omits it, requests guesses from the headers or the content and may guess wrong, producing mojibake like café instead of café. The fix is to set response.encoding = "utf-8" (or the real one) before reading response.text, or to pass response.content (bytes) to BeautifulSoup and let it detect the encoding from the HTML itself.

Python Mojibake and the fix

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When you hand BeautifulSoup bytes instead of text, it reads the <meta charset> and decodes correctly. That is why many scrapers use BeautifulSoup(response.content, "lxml").

Cookies

HTTP itself is stateless: every request stands alone. Cookies add memory. A server sends Set-Cookie: session=abc123; Path=/; HttpOnly, the client stores it and sends Cookie: session=abc123 with later requests to the same site. Cookies are used for:

  • Sessions and logins: proving you are the same visitor.
  • Preferences: language, currency, cookie-consent choices.
  • Tracking: an identifier across visits.

For scrapers, the practical points are:

  • A requests.Session stores and re-sends cookies automatically.
  • Some sites only serve real content after a first visit sets a cookie; a fresh client without it may see a consent wall or a redirect loop.
  • Cookie attributes (Domain, Path, Expires, Secure, HttpOnly, SameSite) control where and how long a cookie is sent. HttpOnly cookies are invisible to page JavaScript but still travel with requests.

Python's http.cookies module can parse a Set-Cookie header, which shows what the client library stores:

Python Parse a Set-Cookie header

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Never copy someone else's session cookie into your scraper, and do not scrape behind a login you are not entitled to use. The legal lesson explains why.

Check your understanding

Quick check · 4 questions

of 4 answered You scored / 4 · perfect!

  1. 1. A server replies 429 with a Retry-After: 30 header. What is the right reaction?

    429 means you are sending too many requests. Retry-After says how long to wait.
  2. 2. What does a 304 Not Modified response tell you?

    After a conditional request, 304 means nothing changed and you can reuse what you saved.
  3. 3. Your text shows 'café' instead of 'café'. What is the most likely cause?

    That pattern appears when UTF-8 bytes are decoded as Latin-1 or a similar single-byte encoding.
  4. 4. Why use a Session object when scraping a site that sets cookies?

    A Session keeps cookies between requests, so the site sees one continuous visitor.

Summary

  • Status code families: 2xx success, 3xx redirect, 4xx you, 5xx server. React differently to each.
  • 403 and 429 are signals about your behavior: slow down, respect Retry-After, and reconsider the approach.
  • Know the key request and response headers, especially User-Agent, Content-Type, Location, Retry-After and Set-Cookie.
  • Conditional requests with ETag skip unchanged pages.
  • Pass bytes to BeautifulSoup to avoid encoding mistakes.
  • Cookies give HTTP memory; a Session handles them for you.

Next: HTML Structure for Scrapers