Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 39 ยท Requests and APIs

Sessions, Cookies and Login

Understand cookies and sessions, keep state across requests, extract CSRF tokens from login forms, and know when logging in is not allowed. Runnable offline demos.

HTTP has no memory. Every request is a fresh conversation, so websites invented a trick to recognize returning visitors: the cookie. The server sends a small named value, the client sends it back on every later request, and from that the server knows "this is the same visitor". Logins, shopping carts, language choices and consent banners all ride on cookies.

For a scraper this matters in two ways. Some sites will not show you any content until you have received a cookie on a first visit. And pages behind a login need you to send credentials and then carry the resulting session cookie with every later request. This lesson explains how cookies work, how to inspect them in Python, how a login form flow looks in code, and, honestly, when you should not log in at all.

How cookies travel

sequenceDiagram
  participant C as Your scraper
  participant S as Server
  C->>S: GET /home
  S-->>C: 200 + Set-Cookie: sid=abc123; HttpOnly; Path=/
  C->>S: GET /account (Cookie: sid=abc123)
  S-->>C: 200 your account page

The server sets a cookie once; the client repeats it on later requests.

A Set-Cookie header carries a name, a value and attributes such as Path, Domain, Expires, Secure and HttpOnly. The Python standard library can parse one, which is handy when you inspect headers from DevTools:

Python Parse Set-Cookie headers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Cookies in a requests Session

A requests.Session stores cookies in a jar and sends the right ones automatically, matching them to the domain and path. You can also put cookies in by hand, for example values you copied from your own browser during development:

Python A session cookie jar

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The cookie for elsewhere.test is correctly left out, because cookies are scoped to the domain that set them. In practice you rarely set cookies yourself: you let the session collect them from Set-Cookie headers.

The login form flow

Most classic logins work like this: you GET the login page, which contains a hidden CSRF token in the form, then you POST the username, password and that token. The server answers with Set-Cookie, and the session is now authenticated. Here is the token-extraction step on a sample form:

Python Extract the hidden fields from a login form

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Collecting every named input and then filling in only the visible ones is more robust than hard-coding field names. The full flow in code (read-only, because it needs a real server):

Python
import requests
from bs4 import BeautifulSoup

BASE = "https://shop.example"
session = requests.Session()

page = session.get(f"{BASE}/login", timeout=10)
soup = BeautifulSoup(page.text, "lxml")
form = soup.select_one("form")
payload = {i["name"]: i.get("value", "") for i in form.select("input[name]")}
payload.update(username=USERNAME, password=PASSWORD)   # read from environment variables

resp = session.post(f"{BASE}{form['action']}", data=payload, timeout=10)
resp.raise_for_status()

if "Sign out" not in resp.text:
    raise RuntimeError("Login failed")

profile = session.get(f"{BASE}/account", timeout=10)

Never hard-code passwords in source files. Read them from environment variables (os.environ["SHOP_PASSWORD"]) or a secrets manager, and keep them out of version control.

Checking that you are really logged in

A failed login often returns 200 OK with the form shown again. Always check for a positive sign: a "Sign out" link, the username on the page, or a redirect to the account page. Without that check your scraper quietly collects the login page hundreds of times.

Saving and reusing a session

Logging in on every run is slow and may trigger security alerts. Save the cookies and load them next time:

Python
import json, requests

with open("cookies.json", "w") as f:
    json.dump(requests.utils.dict_from_cookiejar(session.cookies), f)

# Later
session = requests.Session()
with open("cookies.json") as f:
    requests.utils.add_dict_to_cookiejar(session.cookies, json.load(f))

Treat that file like a password: anyone holding it can act as you.

When not to log in

  • Terms of service. Many sites forbid automated access to logged-in areas. Logging in means you have accepted their terms, so breaking them is a much stronger claim against you than scraping public pages.
  • Other people's data. Behind a login there is often personal data. Collecting it can bring privacy law (GDPR and similar) into play.
  • Your own data. Exporting your own orders or statements is usually fine, and many sites offer an official export that is better than scraping.
  • Two-factor authentication, CAPTCHAs and "are you a robot" checks are explicit signals that automation is not wanted. Do not try to defeat them.

If the content requires an account and the terms forbid automation, look for an official API or ask for permission. Details in Legal and ethical scraping.

Other state: headers and tokens

Cookies are not the only state. Single-page apps often send a bearer token in an Authorization: Bearer ... header, obtained from a login API call. The pattern is the same: log in once, store the token, put it in session.headers, and handle expiry by logging in again when you receive a 401.

Python
session.headers["Authorization"] = f"Bearer {token}"

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a server send to ask the client to store a cookie?

    Set-Cookie travels from server to client; the client answers with a Cookie header on later requests.
  2. 2. Why does the login example copy every named input from the form, including hidden ones?

    Servers use hidden fields such as a CSRF token to validate the form submission.
  3. 3. A login request returns status 200. What should you still check?

    Many sites answer 200 with the login form again when credentials are wrong.
  4. 4. Where should a script read a password from?

    Keep secrets out of source code and version control.

Summary

  • Cookies give HTTP a memory; a Session stores and resends them for you.
  • A login is usually: GET the form, copy hidden fields, POST credentials, verify success.
  • Keep passwords in environment variables and treat saved cookie files as secrets.
  • Logged-in scraping carries extra legal and ethical weight; check the terms first, and prefer official APIs or exports.
  • Do not try to bypass 2FA or CAPTCHAs.

Next: Asynchronous Scraping