Skip to content
Tutorials Center
My learning

Topics

Lesson 101 of 149 ยท Data and the Web

Networking: URLs, HTTP and Sockets

Learn how Python programs talk over the network. Parse and build URLs with urllib.parse, understand HTTP, call APIs with urllib and requests, and meet sockets.

Weather apps, chat bots, price trackers and data pipelines all have one thing in common: they talk to other computers. Python is one of the most popular languages for this kind of work, because its standard library and its ecosystem make network code short and readable.

In this lesson you'll take the network apart layer by layer. You'll start with URLs, the addresses of everything on the web, then learn how HTTP requests and responses are shaped, call web APIs with urllib.request and the popular requests library, and finally look at sockets, the low-level connections everything else is built on.

No network in these editors. For safety, the Python editors on this site can't open network connections. The URL and HTTP-message examples run live, because they only process text. Code that really connects to a server is shown as read-only snippets for you to run on your own computer.

The anatomy of a URL

A URL packs several pieces of information into one string:

TEXT
https://shop.example.com:8443/search/books?q=python+tips&page=2#results
\___/   \______________/ \__/\___________/ \__________________/ \_____/
scheme        host       port     path             query          fragment
  • scheme: the protocol, usually https (or http, ftp, mailto ...).
  • host: the server's name or IP address.
  • port: optional; https uses 443 and http uses 80 unless a port is given.
  • path: which resource on the server.
  • query: extra name=value parameters, separated by &.
  • fragment: a position inside the page; browsers never send it to the server.

The urllib.parse module splits URLs into these parts, so you never have to pick them apart with string methods or regular expressions:

Python Taking a URL apart

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

parse_qs() returns a dictionary of lists, because a parameter can appear more than once (here, tag). It also decodes the query: the + in python+tips means a space. Values always come back as strings, so convert numbers yourself. If you prefer a list of pairs that keeps the original order, use parse_qsl().

Building URLs safely

URLs can only contain a limited set of characters. Spaces, accents, &, ? and # must be percent-encoded (%20 for a space, %C3%A9 for รฉ). Gluing strings together with + or f-strings breaks the moment a value contains one of those characters, and it can even let a user inject extra parameters.

Let urllib.parse do the encoding:

Python Encoding queries and joining links

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

  • urlencode(dict) builds a query string from a dictionary, encoding every value. Notice that the & inside the search text became %26, so it can't be mistaken for a separator.
  • quote() encodes a path segment. By default it leaves / alone; pass safe="" to encode slashes too, for example when a file name goes inside one segment.
  • urljoin(base, link) resolves a relative link the way a browser does. It's essential when you follow links found in a web page.

How HTTP works

HTTP (and its encrypted form, HTTPS) is the language browsers, apps and APIs use to talk to web servers. Every exchange is one request and one response, and both are mostly plain text:

TEXT
GET /v1/weather?city=Oslo HTTP/1.1
Host: api.example.com
Accept: application/json
User-Agent: weather-script/1.0

The first line has the method, the path and the protocol version. Then come headers, one per line, and an empty line. Requests that send data (such as POST) put a body after the empty line.

The most common methods are:

Method Meaning Typical use
GET Read a resource Load a page, fetch data from an API
POST Create something or submit data Send a form, create an order
PUT / PATCH Replace / partly update a resource Edit a profile
DELETE Remove a resource Delete a comment

The response starts with a status line containing a three-digit status code. The first digit tells you the category, and the http module knows them all:

Python HTTP status codes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

HTTPStatus members are integers, so you can compare them with plain numbers. Writing HTTPStatus.TOO_MANY_REQUESTS instead of 429 makes code easier to read.

Reading a response by hand

Libraries parse responses for you, but seeing the raw text once makes everything clearer. This example takes a complete HTTP response, splits off the status line and headers, and decodes the JSON body:

Python Parsing a raw HTTP response

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Lines in HTTP end with \r\n, and header names are case-insensitive, which is why the code lowercases them. Most web APIs answer with JSON, which you already know how to handle from Working with JSON.

Making requests with urllib.request

The standard library can send HTTP requests on its own with urllib.request. Run this on your computer:

Python
import json
from urllib.error import HTTPError, URLError
from urllib.parse import urlencode
from urllib.request import Request, urlopen

url = "https://api.example.com/v1/weather?" + urlencode({"city": "Oslo"})
request = Request(url, headers={"Accept": "application/json",
                                "User-Agent": "weather-script/1.0"})

try:
    with urlopen(request, timeout=10) as response:      # always set a timeout
        print(response.status, response.headers["Content-Type"])
        data = json.load(response)                      # parse the JSON body
except HTTPError as error:          # the server answered with 4xx or 5xx
    print("Server said:", error.code, error.reason)
except URLError as error:           # no answer at all: DNS, refused, offline...
    print("Could not connect:", error.reason)

urlopen() returns a response object that works as a context manager and a file: .read() returns the body as bytes, and json.load() can read it directly. Note the two kinds of failure: HTTPError means the server replied with an error status, while URLError means you never got a reply.

The requests library

urllib.request works, but most Python developers use requests, a third-party package with a friendlier interface. Install it in your virtual environment with pip install requests:

Python
import requests

response = requests.get(
    "https://api.example.com/v1/weather",
    params={"city": "Oslo", "units": "metric"},   # encoded for you
    headers={"Accept": "application/json"},
    timeout=10,
)
response.raise_for_status()           # raises an exception for 4xx and 5xx
weather = response.json()             # parse the JSON body
print(weather["temperature"])

# Sending JSON, for example to create something
new_note = requests.post(
    "https://api.example.com/v1/notes",
    json={"title": "Buy milk", "done": False},    # sets Content-Type for you
    headers={"Authorization": "Bearer <your-token>"},
    timeout=10,
)
print(new_note.status_code)           # 201 Created, if all went well

# A session reuses connections and shared headers across many requests
with requests.Session() as session:
    session.headers["User-Agent"] = "weather-script/1.0"
    for city in ["Oslo", "Lima", "Nairobi"]:
        reply = session.get("https://api.example.com/v1/weather",
                            params={"city": city}, timeout=10)
        print(city, reply.json()["temperature"])

For async programs, httpx offers almost the same interface with async/await support (see Async Programming with asyncio).

Network code checklist. Always pass a timeout, or a silent server can hang your program forever. Check the status code (or call raise_for_status()) before trusting the body. Keep HTTPS certificate checks on: never pass verify=False to "fix" an error. Keep API keys out of your code and load them from environment variables.

Sockets: the layer underneath

Every HTTP library sits on top of sockets. A socket is one end of a two-way connection between programs, identified by an IP address and a port number. TCP sockets deliver a reliable, ordered stream of bytes (used by HTTP, email and databases). UDP sockets send individual messages with no delivery guarantee (used by DNS, games and video calls).

Here is HTTP done by hand over a TCP socket. Run it on your computer to see the raw response from a real server:

Python
import socket

with socket.create_connection(("example.com", 80), timeout=10) as connection:
    request = ("GET / HTTP/1.1\r\n"
               "Host: example.com\r\n"
               "Connection: close\r\n"
               "\r\n")
    connection.sendall(request.encode("ascii"))   # sockets send bytes, not str

    chunks = []
    while chunk := connection.recv(4096):          # an empty bytes object = closed
        chunks.append(chunk)

print(b"".join(chunks)[:300].decode("utf-8", errors="replace"))

And the other side, a tiny server that echoes back whatever a client sends:

Python
import socket

with socket.create_server(("127.0.0.1", 5000)) as server:
    print("Listening on port 5000...")
    while True:
        client, address = server.accept()          # wait for a connection
        with client:
            data = client.recv(1024)
            client.sendall(b"echo: " + data)

You'll rarely write socket code for web work, since libraries handle the details. It's useful for custom protocols, games, and understanding what libraries are doing. Two lessons carry over everywhere: data on the wire is bytes (encode and decode text at the edges), and recv() can return less than you asked for, so you read in a loop.

Addresses themselves are easy to inspect with the ipaddress module, which works without any network:

Python Inspecting IP addresses

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

127.0.0.1 (and ::1 in IPv6) is the loopback address: it always means "this computer", which is why local development servers listen on it. Private ranges such as 192.168.x.x and 10.x.x.x are used inside home and office networks and aren't reachable from the internet directly.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does parse_qs("tag=a&tag=b&page=2") return?

    parse_qs returns a dict of lists of strings, because each parameter may appear several times. parse_qsl returns the list of pairs.
  2. 2. Why should you build query strings with urlencode() instead of an f-string?

    A value containing & or # would otherwise be read as a separator or fragment. urlencode encodes it safely.
  3. 3. An API answers with status 404. What does that mean?

    4xx codes are client errors, and 404 means Not Found. 5xx codes are server errors, and 204 means success with no content.
  4. 4. Which mistake can make network code hang forever?

    Without a timeout, a server that never answers keeps your program waiting indefinitely. Pass timeout= to urlopen, requests and socket calls.
  5. 5. What do socket.sendall() and socket.recv() work with?

    Sockets move raw bytes. Encode text with .encode() before sending and decode received bytes with .decode().

Summary

  • urlsplit() and parse_qs() take URLs apart; urlencode(), quote() and urljoin() build them safely.
  • HTTP is a request/response protocol: a method, path and headers go out; a status code, headers and a body come back.
  • Status codes: 2xx success, 3xx redirect, 4xx client error, 5xx server error; http.HTTPStatus names them all.
  • urllib.request sends requests with only the standard library; the requests package (and async httpx) make it friendlier.
  • Always use timeouts, check status codes, keep certificate checks on and keep secrets out of code.
  • Sockets are the low-level layer: TCP connections between addresses and ports that carry bytes.

Next: Databases with sqlite3, where you'll store data in a real SQL database with nothing to install.