Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 39 ยท HTTP and HTML Basics

URLs and Query Strings

Learn how URLs are built, how to parse them, resolve relative links with urljoin, build query strings safely, normalize URLs and spot pagination patterns.

A scraper lives and dies by URLs. You need to build them (page 2, page 3, a search for "lamp"), take them apart (which site is this link on?), turn relative links into absolute ones, and avoid fetching the same page twice under slightly different spellings. Python's urllib.parse module does all of this, and it needs no network, so every example below runs in your browser.

Parts of a URL

TEXT
https://[email protected]:8443/gadgets/lamps?color=black&page=2#reviews
\___/   \__/ \_______________/ \__/ \____________/ \_______________/ \_____/
scheme  user      host        port      path           query         fragment
  • Scheme: http or https (also ftp, file, mailto).
  • Host (netloc): the server name, with optional user and port.
  • Path: which resource on the server.
  • Query string: key=value pairs after ?, joined by &. Used for search terms, filters, sorting and pagination.
  • Fragment: after #. It is handled by the browser and is never sent to the server, so two URLs that differ only by fragment return the same page.
Python Take a URL apart

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

parse_qs returns each value as a list, because a key may repeat (?tag=a&tag=b). That is why we use params["page"][0].

Building query strings safely

Never glue strings together by hand. Characters like spaces, &, =, + and non-ASCII letters must be percent-encoded (a b becomes a%20b or a+b). urlencode handles it:

Python Encode query parameters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

doseq=True expands lists into repeated keys. With requests you rarely call urlencode yourself, because requests.get(url, params={...}) does it for you:

Python
import requests

r = requests.get(
    "https://shop.example.com/search",
    params={"q": "desk lamp & shade", "page": 2},
    timeout=10,
)
print(r.url)   # https://shop.example.com/search?q=desk+lamp+%26+shade&page=2

Links in HTML are often relative: href="/gadgets/lamp", href="lamp", href="../about", href="?page=3". You must resolve them against the page's own URL before requesting them. urljoin follows the same rules a browser does.

Python Resolve relative links

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Study the results. A path starting with / replaces the whole path; a bare name replaces only the last segment; .. climbs a level; an absolute URL wins outright. If the page has a <base href> tag, resolve against that instead.

If you extract links with a["href"] and request them directly, you will get MissingSchema errors for relative links. Always pass them through urljoin(response.url, href). Use response.url rather than the URL you requested, because redirects may have changed it.

Normalizing URLs to avoid duplicates

The same page can be reached by many spellings:

  • https://Example.com/a and https://example.com/a
  • /a?utm_source=news and /a
  • /a#section and /a
  • /a?b=2&a=1 and /a?a=1&b=2
  • /a/ and /a

A crawler that treats these as different pages downloads the same content repeatedly. Normalize every URL before storing or fetching it.

Python A URL normalizer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Be careful with normalization: removing a query parameter is only safe if the site ignores it. Tracking parameters are generally safe to drop; id=5 is not.

Pagination patterns

Sites split long lists across pages in a few common ways. Recognizing the pattern tells you how to loop.

Pattern Example How to scrape
Page number ?page=3 or /page/3/ Increase the number until you get an empty page or 404
Offset and limit ?offset=40&limit=20 Add limit to the offset each time
Next link <a rel="next" href="..."> Follow the link until it disappears
Cursor token ?cursor=eyJpZCI6... Read the token from each response and send it with the next request
Infinite scroll More items load as you scroll Find the JSON request behind it (chapter 6)

Generate page URLs with a loop and stop on a clear signal rather than a guessed number:

Python
import requests
from urllib.parse import urljoin

page = 1
while True:
    r = requests.get("https://shop.example.com/gadgets", params={"page": page}, timeout=10)
    if r.status_code == 404:
        break
    soup = BeautifulSoup(r.content, "lxml")
    items = soup.select("div.product")
    if not items:
        break
    # ... extract items ...
    page += 1

Add a polite delay between iterations and a maximum page count as a safety net against endless loops.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does parse_qs('a=1&a=2&b=3') return?

    parse_qs always returns lists of values, because a key can repeat.
  2. 2. Which call resolves href='../about' against a page URL?

    urljoin applies the same resolution rules as a browser for relative links.
  3. 3. Is the #fragment part of a URL sent to the server?

    Fragments are handled client-side, so URLs differing only by fragment return the same page.
  4. 4. Why normalize URLs in a crawler?

    Case, trailing slashes, tracking parameters, parameter order and fragments can all produce duplicate fetches if left alone.

Summary

  • A URL has scheme, host, path, query and fragment; urlparse splits it and urlunparse rebuilds it.
  • Build queries with urlencode or requests' params=, never by string concatenation.
  • Resolve relative links with urljoin(response.url, href).
  • Normalize URLs (case, slashes, tracking parameters, parameter order, fragments) to avoid duplicates.
  • Pagination comes as page numbers, offsets, next links or cursors; stop on a clear signal and cap the loop.

Next: BeautifulSoup Basics