Skip to content
Tutorials Center
My learning

Topics

Lesson 118 of 149 ยท Data and the Web

Crawling with Scrapy

Learn web crawling with Scrapy. Write spiders, extract data with CSS and XPath selectors, follow links and pagination, clean items in pipelines, and crawl politely.

In Web Scraping with Python you fetched one page with requests and picked it apart with Beautiful Soup. That works well for a handful of pages. But what if you need ten thousand product pages, spread over hundreds of category pages with "next" buttons? You need to follow links, avoid visiting the same URL twice, run many downloads at once, retry failures, slow down when the site asks you to, and save the results somewhere. That is a crawler, and Scrapy is the framework Python programmers use to build one.

Scrapy is an asynchronous framework: you describe what to extract and which links to follow, and Scrapy's engine handles the downloading, scheduling and concurrency. This lesson teaches the model with real, runnable code where the browser allows it. Scrapy itself needs network sockets and the Twisted engine, so it cannot run on this page. Its selector library, parsel, runs fine, and you will use it for real. For the crawl loop we build a tiny stand-in so you can see exactly what Scrapy does for you.

Only crawl sites you are allowed to crawl. Read the site's terms and its robots.txt, identify your crawler honestly, keep the request rate low and never collect personal data you have no right to use. See the "Scraping politely" section in the web scraping lesson.

What a Scrapy project looks like

Scrapy ships a command line tool that creates the project skeleton:

Terminal
pip install scrapy
scrapy startproject bookshop
cd bookshop
scrapy genspider books books.example.com

You get a folder with settings.py (all configuration), items.py (data definitions), pipelines.py (cleaning and saving) and a spiders/ folder. The spider is the class you write. It names the start URLs and says how to turn a response into items and further requests.

Python
# bookshop/spiders/books.py  (read-only here: it needs a network and Scrapy)
import scrapy

class BooksSpider(scrapy.Spider):
    name = "books"
    start_urls = ["https://books.example.com/catalog/"]

    def parse(self, response):
        for card in response.css("article.book"):
            yield {
                "title": card.css("h2 a::text").get(),
                "price": card.css(".price::text").get(),
                "url": response.urljoin(card.css("h2 a::attr(href)").get()),
            }

        next_page = response.css("a.next::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)
Terminal
scrapy crawl books -O books.json     # write all items to books.json

Two ideas carry the whole framework. First, parse is a generator: it yields items (the data you want) and requests (more pages to fetch). Second, you never loop over pages yourself. Each yielded request goes into Scrapy's scheduler, which downloads it and calls the callback with the response.

Selectors: CSS and XPath, for real

response.css(...) and response.xpath(...) come from parsel, a separate library you can use on its own with any HTML string. A selector query returns a list-like object: .get() gives the first match (or None), .getall() gives every match as a list. Add ::text to read text and ::attr(name) to read an attribute.

Python Extract data with parsel selectors

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

CSS selectors are shorter and read naturally if you know CSS. XPath is more powerful when you need to filter by text, climb to a parent (..) or look at siblings. Use CSS by default and XPath when CSS cannot say it. The browser developer tools (right-click, Inspect, then copy selector) help you find the first draft of a selector, but shorten it: long auto-generated chains break when the page changes.

Inside a loop, always query relative to the card (card.css(...)), not the whole page. For XPath, start relative queries with a dot: card.xpath(".//a/@href"). Without the dot, XPath searches the entire document and you get the first card's data every time.

How a crawl works: a miniature engine

Here is the loop Scrapy runs for you, written out with a dictionary standing in for the internet. It shows the scheduler, the duplicate filter, callbacks and pipelines in about 60 lines. Selectors are the real parsel.

Python A tiny crawler that behaves like Scrapy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check the numbers. Book 1 appears on two catalog pages but is fetched only once, because the duplicate filter remembers every URL. Book 3 has no numeric price, so the pipeline drops it. Book 4 returns a 404 and the crawl simply goes on. In real Scrapy the same things happen, with four key differences: requests run concurrently (16 at a time by default), the scheduler can be persisted to disk, failures can be retried automatically, and every stage has hooks.

Items and item pipelines

You may yield plain dictionaries, or define an item class so typos in field names are caught:

Python
# items.py
import scrapy

class BookItem(scrapy.Item):
    title = scrapy.Field()
    price = scrapy.Field()
    url = scrapy.Field()

Modern Scrapy also accepts dataclass items. After the spider yields an item, it passes through item pipelines, classes with a process_item method that run in the order you set. Typical jobs are cleaning, validation, removing duplicates and saving to a database.

Python
# pipelines.py
from scrapy.exceptions import DropItem

class PricePipeline:
    def process_item(self, item, spider):
        text = item["price"].strip().lstrip("$")
        try:
            item["price"] = float(text)
        except ValueError:
            raise DropItem(f"No numeric price for {item['title']}")
        return item
Python
# settings.py
ITEM_PIPELINES = {
    "bookshop.pipelines.PricePipeline": 300,    # lower numbers run first
}

Item loaders (ItemLoader with input and output processors) are a more declarative way to clean fields, and scrapy crawl books -O out.csv can export JSON, CSV or XML without any code.

Three helpers cover most navigation:

  • response.follow(href, callback): makes a request, resolving relative URLs for you. It also accepts a selector or an <a> element.
  • response.follow_all(css="a.book", callback=...): one call for every matching link.
  • response.urljoin(href): turns a relative link into a full URL (for storing in an item).

To pass data from a list page to a detail page, use cb_kwargs:

Python
def parse(self, response):
    for card in response.css("article.book"):
        yield response.follow(
            card.css("a::attr(href)").get(),
            callback=self.parse_detail,
            cb_kwargs={"category": card.css(".tag::text").get()},
        )

def parse_detail(self, response, category):
    yield {"title": response.css("h1::text").get(), "category": category}

Pagination by "next" link (as above) is the easiest. If a site uses ?page=N, either follow the next link or generate the URLs yourself in start_requests. If the data arrives from a JSON API that the page loads with JavaScript (check the Network tab in your browser tools), call that API directly with response.json(); it is faster and more reliable than parsing HTML.

Crawling politely and surviving the real web

The defaults you most often change live in settings.py:

Python
BOT_NAME = "bookshop"
USER_AGENT = "bookshop-crawler (+https://example.com/contact)"   # say who you are

ROBOTSTXT_OBEY = True            # skip what robots.txt forbids (on by default in new projects)
DOWNLOAD_DELAY = 1.0             # seconds between requests to the same site
CONCURRENT_REQUESTS_PER_DOMAIN = 4

AUTOTHROTTLE_ENABLED = True      # slow down when the server slows down
AUTOTHROTTLE_START_DELAY = 1.0

RETRY_TIMES = 2                  # retry network errors and 5xx a couple of times
HTTPCACHE_ENABLED = True         # cache responses on disk while developing
FEEDS = {"books.jsonl": {"format": "jsonlines"}}

HTTPCACHE_ENABLED is a gift while you write selectors: pages are downloaded once and re-read from disk afterwards, so you do not hammer the site. While experimenting, scrapy shell "https://example.com" opens an interactive prompt with a ready response object where you can try selectors live.

Know the limits. Scrapy downloads HTML; it does not run JavaScript. Pages that build their content in the browser need either their underlying JSON API, or a rendering tool such as Playwright (via the scrapy-playwright plugin) at the price of much lower speed. Sites with login walls, CAPTCHAs and aggressive bot detection are often a sign that crawling is not welcome; check whether the site offers an official API or data export first.

Scrapy or requests plus Beautiful Soup?

Task Better tool
A few pages, one-off script requests + Beautiful Soup
Hundreds to millions of pages Scrapy
Need retries, throttling, caching, exports Scrapy
A single JavaScript-heavy page A browser tool such as Playwright
Data offered through an API The API, with requests or httpx

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does a Scrapy spider's parse method typically yield?

    parse is a generator. Yielded items go to the pipelines and yielded requests go to the scheduler, which downloads them and calls the chosen callback.
  2. 2. What is the difference between .get() and .getall() on a selector result?

    Use .get() when you expect one value (and pass default= to avoid None), .getall() when you want every match.
  3. 3. Inside a loop over cards, why write card.xpath(".//a/@href") with a leading dot?

    A path starting with // is absolute, so every card would return the first match on the whole page. .// means 'inside this element'.
  4. 4. What does Scrapy's duplicate filter do?

    Pages linked from many places (such as the same book on several list pages) are fetched once, which saves time and avoids loading the server needlessly.
  5. 5. A site shows its data only after JavaScript runs. What is a good first step before reaching for a browser tool?

    Scrapy does not execute JavaScript, but the data often comes from a JSON endpoint you can call directly, which is faster and more reliable than rendering the page.

Summary

  • Scrapy crawls many pages concurrently: you write a spider whose parse generator yields items and requests, and the engine does downloading, scheduling and de-duplication.
  • Extract data with CSS (::text, ::attr()) or XPath selectors from parsel; use .get() / .getall() and always query relative to the current element.
  • Item pipelines clean, validate, drop and store items; DropItem rejects bad records.
  • response.follow, follow_all and cb_kwargs handle links, pagination and passing data between callbacks.
  • Settings control politeness: ROBOTSTXT_OBEY, DOWNLOAD_DELAY, AutoThrottle, retries, an honest USER_AGENT and the HTTP cache while you develop.
  • Scrapy does not run JavaScript: prefer the site's JSON API, and for small jobs requests plus Beautiful Soup is simpler.

Next: Image Processing with Pillow.