Skip to content
Tutorials Center
My learning

Topics

Lesson 3 of 39 ยท Getting Started

Your First Scraper

Build a complete web scraper in Python from fetching to saving: download a page with requests, parse it with BeautifulSoup, extract products and write them to CSV.

In this lesson you will build a complete scraper from start to finish. First you will see the network code you would run on a real site (shown read-only, because the in-browser editors have no internet access), then you will run the parsing and saving steps on a page held in a string. The two halves connect through one value: the HTML text.

The target is a small, imaginary shop page. The structure mirrors what you find on real sites, so the same approach carries over.

Step 1: fetch the page

With requests, fetching a page takes two lines. This snippet is read-only because it needs a network:

Python
import requests

url = "https://example.com/shop"
headers = {"User-Agent": "price-tracker-demo/1.0 (contact: [email protected])"}

response = requests.get(url, headers=headers, timeout=10)
response.raise_for_status()      # stop on 404, 500 and similar
html = response.text             # the page as a string

Three habits are worth learning immediately:

  • timeout: without it, a hung server can freeze your script forever.
  • raise_for_status(): turns error responses into exceptions instead of silently parsing an error page.
  • A descriptive User-Agent: tells the site owner who is visiting and how to reach you. The default python-requests/x.y value is often blocked.

Chapter 5 explains each of these in depth. For now, treat html as "the page source".

Step 2: parse

Here is the page source we will work with. In a real scraper this string would be response.text.

Python
html = """
<html>
<head><title>Corner Shop - Gadgets</title></head>
<body>
  <h1>Gadgets</h1>
  <div class="product" data-id="101">
    <h2 class="name">USB-C Cable</h2>
    <p class="price">$7.99</p>
    <p class="stock">In stock</p>
    <a href="/gadgets/usb-c-cable">Details</a>
  </div>
  <div class="product" data-id="102">
    <h2 class="name">Desk Lamp</h2>
    <p class="price">$24.50</p>
    <p class="stock">Out of stock</p>
    <a href="/gadgets/desk-lamp">Details</a>
  </div>
  <div class="product" data-id="103">
    <h2 class="name">Phone Stand</h2>
    <p class="price">$11.00</p>
    <p class="stock">In stock</p>
    <a href="/gadgets/phone-stand">Details</a>
  </div>
</body>
</html>
"""

Parsing is one line: soup = BeautifulSoup(html, "lxml"). The result, soup, is a tree of Python objects you can search.

Step 3: extract

Look at the page structure. Each product is a div with class product, and inside it sit a name, a price, a stock note and a link. The plan is to find all the product blocks, then read the same four fields from each one.

Python Extract the products

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Study what happened:

  • find_all("div", class_="product") returned a list of the three blocks. The trailing underscore in class_ avoids Python's reserved word class.
  • block["data-id"] read an attribute like a dictionary key.
  • get_text(strip=True) returned the visible text without surrounding whitespace.
  • We converted the strings into proper types: an int, a float and a bool. Clean types now make later analysis easier.

Searching inside block instead of soup is important. block.find("p", class_="price") looks only within that product, so each product gets its own price. Searching the whole soup would return the first price on the page every time.

Step 4: store

Python's standard csv module writes the rows. In the editor we write into an in-memory buffer and print it, because the sandbox has no file system you can keep; locally you would open a real file instead.

Python Save the rows as CSV

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

On your computer you replace the buffer with a file:

Python
with open("products.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["id", "name", "price"])
    writer.writeheader()
    writer.writerows(rows)

Using newline="" prevents blank lines on Windows, and encoding="utf-8" keeps accented characters and currency symbols intact.

Putting it together

The finished script is a short function that takes HTML and returns rows. Keeping parsing separate from fetching is a strong design: you can test the parser on saved HTML and reuse it unchanged if you later switch from requests to a browser.

Python
def parse_products(html: str) -> list[dict]:
    soup = BeautifulSoup(html, "lxml")
    return [
        {
            "id": int(b["data-id"]),
            "name": b.find("h2", class_="name").get_text(strip=True),
            "price": float(b.find("p", class_="price").get_text(strip=True).lstrip("$")),
        }
        for b in soup.select("div.product")
    ]

def main():
    response = requests.get(URL, headers=HEADERS, timeout=10)
    response.raise_for_status()
    rows = parse_products(response.text)
    save_csv(rows, "products.csv")

Common first-scraper mistakes

  • Searching soup instead of the block. You get the same value for every item.
  • Forgetting strip=True. Text arrives with newlines and spaces that break comparisons.
  • Assuming every field exists. One product without a price crashes the loop with AttributeError: 'NoneType'. Handling Messy HTML shows how to prevent it.
  • Not checking the result. Always print len(rows). Zero rows usually means the selector is wrong or the page needs JavaScript.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why pass timeout to requests.get?

    Without a timeout a stalled connection can block your script indefinitely.
  2. 2. Inside the loop you wrote soup.find('p', class_='price') instead of block.find(...). What happens?

    Searching the whole soup always finds the first match in the document, so searching inside the current block is essential.
  3. 3. What does block['data-id'] return?

    A tag behaves like a dictionary of its attributes, and values are strings, so convert with int() if needed.
  4. 4. Why is it a good design to put parsing in a function that accepts an HTML string?

    Separating fetching from parsing makes the parser testable offline and independent of the fetching tool.

Summary

  • A scraper fetches HTML, parses it, extracts fields from each repeated block and stores the rows.
  • Always use a timeout, raise_for_status() and a descriptive user agent when fetching.
  • Search inside each block, not the whole document, and use get_text(strip=True) plus type conversion.
  • Write rows with csv.DictWriter, using newline="" and UTF-8 when saving a file.
  • Keep parsing separate from fetching so you can test on saved HTML.

Next: Legal and Ethical Basics