Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 39 ยท Crawling, Politeness and Ethics

The Scrapy Framework

Learn how Scrapy structures a crawler: spiders, requests and callbacks, selectors, items, pipelines and polite settings, plus a runnable mini-spider that shows the model.

In the last lessons you built the parts of a crawler by hand: a frontier, a visited set, delays, retries, caching. Scrapy is a framework that gives you all of that, already tested and tuned. You write only the part that is specific to your site: where to start, which links to follow, and what to extract. Scrapy takes care of scheduling, concurrency, retries, throttling, robots.txt and exporting the data.

Scrapy is a big Twisted-based package that needs a real Python environment, so its code is read-only on this page. To understand how it works you do not need to run it. We recreate its core idea, spiders that yield items and requests to an engine, in about forty lines you can run.

Install and create a project

Terminal
pip install scrapy
scrapy startproject bookshop
cd bookshop
scrapy genspider books books.example.com

The generated project contains settings.py, an items.py, a pipelines.py, middlewares.py and a spiders/ folder. You will mostly edit spiders, settings and pipelines.

Your first spider

A spider is a class with a name, start URLs and methods ("callbacks") that receive responses and yield either scraped data or further requests.

Python
import scrapy

class BooksSpider(scrapy.Spider):
    name = "books"
    allowed_domains = ["books.example.com"]
    start_urls = ["https://books.example.com/catalog/"]

    def parse(self, response):
        for card in response.css("article.book"):
            yield {
                "title": card.css("h3 a::attr(title)").get(),
                "price": card.css(".price::text").re_first(r"[\d.]+"),
                "url": response.urljoin(card.css("h3 a::attr(href)").get()),
            }

        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

Run it and export the results in one command:

Terminal
scrapy crawl books -O books.json

(-O overwrites the file, -o appends. The extension picks the format: .json, .jsonl, .csv, .xml.)

Things worth noticing:

  • response.css(...) and response.xpath(...) return selector lists. ::text and ::attr(name) are Scrapy's CSS extensions to extract text and attributes. .get() returns the first match (or None); .getall() returns a list.
  • response.follow(url, callback=...) resolves relative URLs and queues a new request. Following links is just yielding more requests.
  • Yielding a plain dict (or an Item) sends it down the pipeline.
  • Duplicate URLs are filtered out automatically by the scheduler.

The engine model

flowchart LR
  S["Spider"] -- "yields Requests" --> E["Engine"]
  E --> SC["Scheduler (queue + dedupe)"]
  SC --> D["Downloader (+ middlewares)"]
  D -- "Response" --> S
  S -- "yields Items" --> P["Item pipelines"]
  P --> O["Feed export / database"]

This is exactly the loop you wrote yourself in Building a crawler, with each box replaced by a robust component. Downloader middlewares wrap requests and responses (user agents, proxies, retries), and spider middlewares wrap what the spider yields.

A mini-Scrapy you can run

The following stand-in works the same way: a spider yields dicts (items) and Request objects, and a tiny engine keeps a queue, filters duplicates, "downloads" from a fake site, calls the right callback, and passes items through a pipeline function. It is the same structure with the networking taken out.

Python A mini engine: spiders yield items and requests

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real Scrapy does the same dance asynchronously, with many requests in flight, but the programming model for you is identical: yield items, yield requests.

Items, loaders and pipelines

For anything beyond a quick script, declare the fields once:

Python
import scrapy

class BookItem(scrapy.Item):
    title = scrapy.Field()
    price = scrapy.Field()
    url = scrapy.Field()

An item pipeline is a class with process_item(item, spider) that every item passes through, in an order set in settings.py. It is the right place for cleaning, validation, de-duplication and storage:

Python
from scrapy.exceptions import DropItem

class CleanPricePipeline:
    def process_item(self, item, spider):
        raw = item.get("price")
        if raw is None:
            raise DropItem(f"missing price: {item.get('url')}")
        item["price"] = float(raw)
        return item

class SqlitePipeline:
    def open_spider(self, spider):
        import sqlite3
        self.db = sqlite3.connect("books.db")
        self.db.execute("CREATE TABLE IF NOT EXISTS books (url TEXT PRIMARY KEY, title TEXT, price REAL)")

    def close_spider(self, spider):
        self.db.commit(); self.db.close()

    def process_item(self, item, spider):
        self.db.execute("INSERT OR REPLACE INTO books VALUES (?, ?, ?)",
                        (item["url"], item["title"], item["price"]))
        return item

Enable them with ITEM_PIPELINES = {"bookshop.pipelines.CleanPricePipeline": 300, "bookshop.pipelines.SqlitePipeline": 400}; lower numbers run first. More on cleaning and storage in Cleaning scraped data and Saving to CSV, JSON and SQLite.

Politeness settings

Scrapy's defaults are fast. Set these in settings.py before pointing it at a real site:

Python
BOT_NAME = "bookshop"
USER_AGENT = "BookshopBot/1.0 (+https://example.org/bot; [email protected])"
ROBOTSTXT_OBEY = True                   # the generated project enables this
DOWNLOAD_DELAY = 1.0                    # base delay per domain
RANDOMIZE_DOWNLOAD_DELAY = True         # jitter: 0.5x to 1.5x
CONCURRENT_REQUESTS_PER_DOMAIN = 2
AUTOTHROTTLE_ENABLED = True             # adapts to the server's latency
AUTOTHROTTLE_START_DELAY = 1.0
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
RETRY_TIMES = 3
RETRY_HTTP_CODES = [429, 500, 502, 503, 504]
HTTPCACHE_ENABLED = True                # cache pages while developing
DEPTH_LIMIT = 5
CLOSESPIDER_PAGECOUNT = 1000            # hard stop as a safety net

This is the lesson on Rate limiting and retries as a settings file. ROBOTSTXT_OBEY and AUTOTHROTTLE_ENABLED in particular are the two switches every polite spider should have on.

Useful extras

  • CrawlSpider with Rule(LinkExtractor(allow=r"/product/"), callback="parse_item", follow=True) follows links by pattern without writing the loop.
  • Scrapy shell: scrapy shell "https://books.example.com/" opens a Python prompt with response loaded, perfect for trying selectors.
  • Passing data between callbacks: yield response.follow(url, self.parse_detail, cb_kwargs={"category": cat}).
  • JavaScript pages: the scrapy-playwright plugin renders selected requests in a browser while keeping Scrapy's engine (Playwright basics).
  • Stats and logging: at the end of a run Scrapy prints counts of requests, responses by status code, items and errors, which tells you quickly whether a run was healthy.

When to choose Scrapy

Use it when you crawl many pages, need concurrency, retries and throttling out of the box, or want a maintainable structure for a long-lived scraper. For a one-off script that fetches ten pages, requests plus BeautifulSoup is simpler and quicker to write.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In a Scrapy spider, how do you follow a link to another page?

    Spiders yield Requests; the engine schedules them and calls the callback with the response.
  2. 2. What is an item pipeline for?

    Every yielded item passes through the enabled pipelines in order.
  3. 3. Which two settings should a polite spider have enabled?

    Obeying robots.txt and letting AutoThrottle adapt to the server keeps load modest.
  4. 4. What does .get() return in a Scrapy selector chain?

    .get() returns the first result or None; .getall() returns every match.

Summary

  • Scrapy supplies scheduling, de-duplication, concurrency, retries, throttling and exports; you write spiders, items and pipelines.
  • A spider yields items and requests; response.follow is how crawling happens.
  • Pipelines clean, validate and store items; settings control politeness.
  • The model is the same loop you built by hand, so the mini-engine above is a faithful picture of it.
  • Choose Scrapy for large or long-lived crawls and simple scripts for small jobs.

Next: Legal and Ethical Scraping