Python Reference ยท Standard Library: Concurrency, Network and Tools
The urllib.parse module
Reference for Python's urllib.parse: urlparse, urlsplit, urlunparse, urljoin, urlencode, parse_qs, quote and unquote, with examples and common pitfalls.
urllib.parse takes URLs apart, puts them back together, resolves relative links and encodes the special characters that are not allowed in URLs. It works on plain strings and never touches the network, so it is safe to use anywhere.
Reach for it instead of splitting URLs with split("/") or gluing query strings together by hand. Those approaches break on ports, fragments, repeated parameters and anything that needs percent-encoding.
from urllib.parse import urlparse parts = urlparse("https://example.com:8080/docs/intro?lang=en#top") print(parts.hostname, parts.port, parts.path)
Try it
Press Run (or Ctrl + Enter) to see the result here.
Program input ยท what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.
Press Run (or Ctrl + Enter) to see the result here.
Program input ยท what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.
Syntax
urlparse(urlstring, scheme='', allow_fragments=True) urlsplit(urlstring, scheme='', allow_fragments=True) urlunparse(components) urljoin(base, url, allow_fragments=True) urlencode(query, doseq=False, safe='', encoding=None, errors=None, quote_via=quote_plus) parse_qs(qs, keep_blank_values=False, strict_parsing=False, encoding='utf-8', errors='replace', max_num_fields=None) quote(string, safe='/', encoding=None, errors=None)
| Function | Description |
|---|---|
urlparse(url) |
Splits into scheme, netloc, path, params, query, fragment. |
urlsplit(url) |
Like urlparse but without the rarely used params part; usually what you want. |
urlunparse(parts), urlunsplit(parts) |
Reverse of the above. |
urljoin(base, url) |
Resolves a relative link against a base URL, as a browser does. |
urlencode(query) |
Turns a dict or list of pairs into a=1&b=2. |
parse_qs(qs) |
Query string to a dict of lists. |
parse_qsl(qs) |
Query string to a list of (key, value) pairs, keeping order and repeats. |
quote(s), quote_plus(s), unquote(s), unquote_plus(s) |
Percent-encode and decode text. quote_plus turns spaces into +. |
Return value: urlparse and urlsplit return named tuples (ParseResult, SplitResult) with extra attributes hostname, port, username and password.
Raises: ValueError for malformed input such as an invalid IPv6 literal, and when reading .port finds a non-numeric or out-of-range port.
Usage notes
parse_qsalways returns lists, because a key may repeat. Useparams["q"][0]to get a single value.- Blank values are dropped by default; pass
keep_blank_values=Trueto keepc=. hostnameis lowercased and has no port or credentials;netlocis the rawuser:pw@host:porttext.quoteleaves/alone by default. For a single path segment or a query value usesafe="".urlencodepairs with+for spaces (form style). For URL paths usequote, which produces%20.- Non-ASCII text is encoded as UTF-8 bytes first:
cafรฉbecomescaf%C3%A9. urljoinfollows RFC rules: a leading/replaces the whole path, and a trailing-slash difference in the base changes the result, so.../guideand.../guide/resolve differently.- Parsing is not validation.
urlparse("not a url")returns a result without error. Check the scheme and host yourself before trusting a URL, especially before fetching it. - Do not splice user input into a URL with f-strings; build it with
urlencodeandquote.
Related
http.serverandurllib,socketre,pathlib- Lessons: Networking, Web development