Skip to content
Tutorials Center
My learning

Topics

Lesson 2 of 39 ยท Getting Started

Setting Up Your Scraping Toolkit

Install Python, create a virtual environment and meet the main web scraping libraries: requests, BeautifulSoup, lxml, Selenium, Playwright and Scrapy, and when to use each.

You can scrape with almost any language, but Python is the most popular choice because it has excellent libraries for every step and reads almost like plain English. This lesson shows how to prepare a clean project on your own computer, introduces the libraries you will use in this course, and helps you choose the right one for a given job.

The in-browser examples on this site already include BeautifulSoup and lxml, so you can follow every lesson without installing anything. Set up your own machine when you want to scrape real pages.

Install Python and create a virtual environment

Install Python 3.10 or newer from python.org (or through your system package manager) and check it:

Terminal
python --version

Always work inside a virtual environment: an isolated folder holding the packages for one project. It prevents version clashes between projects and keeps your system Python clean.

Terminal
mkdir price-tracker
cd price-tracker
python -m venv .venv

# activate it
source .venv/bin/activate        # macOS and Linux
.venv\Scripts\activate           # Windows (PowerShell or cmd)

pip install requests beautifulsoup4 lxml
pip freeze > requirements.txt

When the environment is active, your prompt usually shows (.venv) and pip install puts packages only into that project. The requirements.txt file records exact versions, so a teammate (or you, in six months) can recreate the setup with pip install -r requirements.txt.

The package is installed as beautifulsoup4 but imported as bs4: from bs4 import BeautifulSoup. This trips up almost every beginner once.

The libraries at a glance

Scraping tools fall into four groups.

Fetching pages

  • requests: the friendly, standard HTTP client for Python. Synchronous and simple. Covered in chapter 5.
  • httpx: similar to requests, with async support and HTTP/2.
  • urllib (standard library): always available, but wordier.

Parsing HTML

  • BeautifulSoup (bs4): forgiving, beginner-friendly navigation and search. The focus of chapter 3.
  • lxml: a fast C-based parser with full XPath support. Chapter 4.
  • html.parser (standard library): the built-in parser BeautifulSoup can use.
  • parsel and selectolax: alternatives when you want selectors with more speed.

Browsers for dynamic pages

  • Selenium: automates a real browser. Old and widely used.
  • Playwright: modern browser automation with auto-waiting. Chapter 6.

Frameworks

  • Scrapy: a full crawling framework with queues, pipelines, throttling and export. Chapter 7 introduces it.
flowchart TD
    Q["Does the data appear<br/>in the page source?"]
    Q -->|"Yes"| S["requests + BeautifulSoup<br/>or lxml"]
    Q -->|"No, JavaScript builds it"| J["Look for a JSON endpoint"]
    J -->|"Found one"| R["requests + json"]
    J -->|"None"| B["Playwright or Selenium"]
    S --> M["Many pages?"]
    R --> M
    B --> M
    M -->|"Hundreds or more"| F["Consider Scrapy"]

Pick the lightest tool that works. A browser is the heaviest option, so use it last.

Check what is available

The editor below runs real Python in your browser. It imports the parsing libraries and prints their versions, the same check you would run after a local install.

Python Check your toolkit

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

If you ran a script like this locally and got an ImportError or ModuleNotFoundError, the library is missing from the active environment. Check that the environment is activated and run pip install again.

Parsers inside BeautifulSoup

BeautifulSoup itself does not read HTML. It wraps a parser that you choose with the second argument:

Python The same HTML, three parsers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All three accept the broken snippet but repair it differently. html.parser keeps it nearly as written, lxml wraps it in html and body, and html5lib builds the tree exactly as a browser would, including an empty head. The tradeoffs:

Parser Speed Leniency Needs install
html.parser Medium Good No
lxml Fast Good Yes
html5lib Slow Browser-exact Yes

Choose lxml for speed in most projects, html.parser when you cannot install anything, and html5lib when a page is so broken that other parsers disagree with what the browser shows.

Browser developer tools

The most important tool is not a Python library at all. Every browser has developer tools (press F12 or Ctrl+Shift+I). Three panels matter:

  • Elements shows the live DOM and lets you copy a CSS selector or XPath for any element (right-click, Copy).
  • Network lists every request the page makes, including hidden JSON calls. Filter by Fetch/XHR.
  • Console lets you test selectors: document.querySelectorAll("li.book").

Get into the habit of exploring in the browser first. Once a selector works there, it will almost always work in your scraper.

A sensible project layout

Keep scraping projects tidy from the start:

TEXT
price-tracker/
  .venv/              # never commit this
  requirements.txt
  scraper.py          # fetch + parse + extract
  data/
    raw/              # saved HTML, so you can re-parse without re-downloading
    products.csv

Saving raw HTML is a professional habit. If you later find a bug in your parser, you can fix it and re-run on the saved files instead of hitting the site again.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why use a virtual environment?

    A virtual environment keeps each project's dependencies separate so versions do not clash.
  2. 2. How do you import BeautifulSoup?

    The package installs as beautifulsoup4 but its import name is bs4.
  3. 3. Which tool should you try first when the data is visible in the page source?

    If the data is already in the HTML, a plain request plus a parser is simpler and much faster than a browser.
  4. 4. Which parser builds the tree most like a real browser does?

    html5lib implements the HTML5 parsing algorithm used by browsers, at the cost of speed.

Summary

  • Use a virtual environment per project and record dependencies in requirements.txt.
  • requests fetches, BeautifulSoup and lxml parse, Playwright and Selenium drive browsers, Scrapy crawls at scale.
  • BeautifulSoup wraps a parser: html.parser, lxml or html5lib.
  • Browser developer tools (Elements, Network, Console) are your most important scraping tools.
  • Save raw HTML so parser fixes do not require new downloads.

Next: Your First Scraper