Python Reference ยท Third-party Libraries
The requests library
Overview of the requests library: get and post, params, headers, JSON bodies, status codes, timeouts, sessions, error handling and common mistakes.
requests is the most popular third-party library for making HTTP calls from Python. It hides the details of the standard library's urllib behind short, readable calls: one line to fetch a page, one line to send JSON, and an easy way to inspect the response.
Install it with python -m pip install requests. The examples on this page are read-only: the in-browser runner has no real network access to run them, and the library is not preloaded. Copy them into a script on your own machine.
import requests response = requests.get("https://api.example.com/items", timeout=10) print(response.status_code)
Basic requests
The snippets use made-up example URLs. Replace them with a service you are allowed to call.
import requests # GET with query parameters: ?page=2&per_page=20 r = requests.get( "https://api.example.com/items", params={"page": 2, "per_page": 20}, headers={"Accept": "application/json"}, timeout=10, ) print(r.status_code) # 200 print(r.ok) # True for status codes below 400 print(r.headers["Content-Type"]) print(r.url) # the final URL including the query string print(r.json()) # parsed body as lists and dicts print(r.text) # body as a string # POST a JSON body r = requests.post( "https://api.example.com/items", json={"name": "notebook", "qty": 3}, timeout=10, ) # POST a form r = requests.post("https://api.example.com/login", data={"user": "ada"}, timeout=10)
Handling errors
import requests try: r = requests.get("https://api.example.com/items", timeout=(3, 10)) r.raise_for_status() # raises HTTPError for 4xx and 5xx data = r.json() except requests.exceptions.Timeout: print("The server took too long") except requests.exceptions.HTTPError as err: print("Bad status:", err.response.status_code) except requests.exceptions.RequestException as err: print("Network problem:", err) # base class of all requests errors except ValueError: print("The body was not valid JSON")
Sessions
A Session keeps cookies and default headers, and reuses the underlying connection, which is faster for many calls to the same host.
import requests with requests.Session() as s: s.headers.update({"Authorization": "Bearer YOUR_TOKEN"}) s.get("https://api.example.com/profile", timeout=10) s.get("https://api.example.com/orders", timeout=10)
Try it
The runner cannot reach the network, so here is a runnable stand-in that shows how code written against a response object is tested without any real server: a tiny fake response class with the same status_code, ok and json() members.
Press Run (or Ctrl + Enter) to see the result here.
Program input ยท what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.
Parameters
| Parameter | Description |
|---|---|
params |
Dict added to the URL as a query string, encoded for you. |
data |
Form fields (application/x-www-form-urlencoded) or raw bytes. |
json |
Python object sent as JSON, with the Content-Type header set. |
headers |
Dict of extra request headers. |
timeout |
Seconds to wait, or a (connect, read) tuple. |
auth |
A (user, password) tuple for basic authentication, or an auth object. |
allow_redirects |
Follow redirects (default True for everything except HEAD). |
stream |
Do not download the body at once; read it in chunks with iter_content(). |
verify |
Check TLS certificates (default True; leave it on). |
Return value: a Response object with status_code, ok, headers, text, content (bytes), json(), url, cookies and raise_for_status().
Usage notes
- Always set a
timeout. The default is to wait forever, so a stalled server freezes your program. - A 404 or 500 is not an exception until you call
raise_for_status(). Checkr.okor the status code yourself. json=anddata=differ.data=dictsends a form,json=dictsends a JSON document. Do not pass both for the same body.r.json()raisesrequests.exceptions.JSONDecodeError(aValueErrorsubclass) when the body is not JSON, for example an HTML error page.- Never put secrets in URLs. Use headers or the
authargument, and keep tokens out of source code. - Retries are not automatic. Mount an
HTTPAdapterconfigured withurllib3'sRetry, or write a small loop with a backoff. - No async support. For many concurrent requests use
asynciowith a library such ashttpxoraiohttp, or threads. - Standard-library alternative:
urllib.requestneeds no installation but takes more code.
Related
urllib.parse,http.serverandurllib,json,socket- Flask and FastAPI
- Lessons: Networking, JSON