Web Scraping Tutorial
Learn Web Scraping step by step: 39 lessons in 9 chapters with examples, quizzes, diagrams and projects.
Start learning →
of lessons completed
Getting Started
- What Is Web Scraping? Learn what web scraping is, how a scraper turns web pages into structured data, when to scrape and when to use an API, and what the typical pipeline looks like.
- Setting Up Your Scraping Toolkit Install Python, create a virtual environment and meet the main web scraping libraries: requests, BeautifulSoup, lxml, Selenium, Playwright and Scrapy, and when to use each.
- Your First Scraper Build a complete web scraper in Python from fetching to saving: download a page with requests, parse it with BeautifulSoup, extract products and write them to CSV.
- Legal and Ethical Basics of Web Scraping Understand the legal and ethical side of web scraping: robots.txt, terms of service, copyright, personal data, rate limits and a practical checklist before you scrape.
HTTP and HTML Basics
- HTTP Requests and Responses Learn how HTTP works for web scrapers: methods, request and response anatomy, headers, bodies and how a browser page load turns into many requests.
- Status Codes, Headers and Cookies Master the HTTP details a scraper depends on: status code families, redirects, 403 and 429 blocks, the headers that matter, content encoding and how cookies work.
- HTML Structure for Scrapers Learn to read HTML the way a scraper must: elements, attributes, the DOM tree, ids and classes, tables and lists, data attributes, and how to inspect a page.
- URLs and Query Strings Learn how URLs are built, how to parse them, resolve relative links with urljoin, build query strings safely, normalize URLs and spot pagination patterns.
Parsing with BeautifulSoup
- BeautifulSoup Basics Start with BeautifulSoup: create a soup object, choose a parser, access tags by name, work with Tag and NavigableString objects, read attributes and print the tree.
- Navigating the Parse Tree Walk the BeautifulSoup tree: children, descendants, parent, parents, next and previous siblings, next_element and a practical label-and-value scraping pattern.
- find() and find_all() in BeautifulSoup Master BeautifulSoup searching: find and find_all by tag, class, id, attributes, text, regular expressions, lists and functions, with limit, recursive and string filters.
- Extracting Text and Attributes Turn tags into clean values: get_text options, strip and separator, attribute extraction, converting prices and dates, handling missing fields and building records.
- Modifying the Tree and Choosing a Parser Clean HTML before extracting: remove scripts and ads with decompose and extract, replace and wrap tags, find text and replace it, and compare html.parser, lxml and html5lib.
Selectors and Data Extraction
- CSS Selectors with select() Use CSS selectors in BeautifulSoup: select and select_one, combinators, attribute selectors, nth-child, :not, :has, :-soup-contains and a selector cheat sheet for scrapers.
- XPath with lxml Learn XPath for scraping: lxml.html parsing, axes, predicates, text() and contains(), attribute extraction, relative paths, and when XPath beats CSS selectors.
- Regular Expressions in Scraping Use Python regular expressions alongside parsers: search, findall, groups, named groups, extracting prices, phone numbers, dates and IDs from text, and when not to use regex.
- Extracting Tables and Lists Turn HTML tables, nested lists and definition lists into Python dictionaries and CSV: headers, colspan and rowspan, multiple tables, and pandas.read_html.
- Extracting Links, Images and Metadata Extract links, images and srcset, page titles, meta and Open Graph tags, canonical URLs and JSON-LD structured data from HTML with BeautifulSoup and the json module.
- Handling Messy HTML and Layout Changes Write scrapers that survive real pages: defensive extraction, fallback selectors, validation, logging, tests with saved HTML and detecting when a site layout changes.
Requests and APIs
- The requests Library Fetch pages with Python requests: GET and params, headers, timeouts, status codes, raise_for_status, sessions and retries. Includes offline examples you can run.
- Finding and Using JSON APIs Skip HTML parsing: find the JSON endpoints a website uses, read them with requests, and extract nested data safely in Python. Runnable offline examples.
- Pagination Patterns Scrape every page of a list: page numbers, offsets, cursors, next links, Link headers and infinite scroll. Reusable Python generators with offline demos.
- Sessions, Cookies and Login Understand cookies and sessions, keep state across requests, extract CSRF tokens from login forms, and know when logging in is not allowed. Runnable offline demos.
- Asynchronous Scraping with asyncio Speed up scraping with asyncio: concurrent fetching, semaphores to limit load, timeouts, error handling and threads, with offline runnable examples.
Dynamic Pages and Browser Automation
- JavaScript-Rendered Pages Why requests sees an empty page, how to check for JavaScript rendering, and how to extract data embedded in script tags, JSON-LD and window state with Python.
- Selenium Basics Control a real browser from Python with Selenium: start headless Chrome, locate elements, read text, click and fill forms, then hand the HTML to BeautifulSoup.
- Playwright Basics Scrape JavaScript sites with Playwright for Python: sync API, locators with auto-waiting, reading content, intercepting API responses and blocking heavy resources.
- Waits, Scrolling and Interactions Handle infinite scroll, load-more buttons, popups and flaky elements in browser automation, with the stop rules and retry patterns that keep scrapers reliable.
Crawling, Politeness and Ethics
- robots.txt and Sitemaps Read and obey robots.txt with urllib.robotparser, honor Crawl-delay, and use XML sitemaps to discover pages the polite way. Runnable Python examples.
- Building a Crawler Build a breadth-first web crawler in Python: a frontier queue, URL normalization, a visited set, same-domain rules, depth limits and resumable state. Runs offline.
- Rate Limiting and Retries Be a good citizen on the web: delays with jitter, token buckets, 429 and Retry-After handling, exponential backoff and caching so you never re-download a page.
- The Scrapy Framework Learn how Scrapy structures a crawler: spiders, requests and callbacks, selectors, items, pipelines and polite settings, plus a runnable mini-spider that shows the model.
- Legal and Ethical Scraping An honest guide to the legality and ethics of web scraping: terms of service, copyright, personal data, access controls, and a practical checklist before you scrape.
Storing and Cleaning Data
- Cleaning Scraped Data Turn messy scraped strings into clean values: whitespace, prices and currencies, dates, numbers, units, missing values and validation, with runnable Python examples.
- Saving to CSV, JSON and SQLite Store scraped data in the right format: CSV with the csv module, JSON and JSON Lines, SQLite with upserts, and pandas exports. Runnable Python examples.
- Deduplicating and Updating Data Remove duplicates from scraped data with stable keys and fuzzy matching, detect new, changed and removed items between runs, and scrape incrementally.
Projects and Practice
- Project: Book Catalog Scraper Build a complete scraper from crawl to report: walk paginated category pages, parse book cards, clean prices and ratings, store in SQLite and analyze the results.
- Project: Price Tracker Build a price-tracking scraper: extract prices from product pages, keep a history in SQLite, detect drops, handle failures and send alerts. Runnable offline project.
- Practice Exercises Eight scraping exercises with runnable solutions: tables, nested lists, JSON, pagination, robots rules, cleaning, deduplication and a mini crawler. Try each before reading the answer.