Skip to content
Tutorials Center
My learning

Topics

Lesson 1 of 39 ยท Getting Started

What Is Web Scraping?

Learn what web scraping is, how a scraper turns web pages into structured data, when to scrape and when to use an API, and what the typical pipeline looks like.

Web scraping is the practice of using a program to read web pages and pull out the pieces of information you care about. Instead of copying prices, headlines or job posts into a spreadsheet by hand, you write a small script that visits each page, finds the data inside the HTML and saves it in a clean, structured form such as a CSV file or a database table.

This course teaches scraping with Python. You will learn to read HTML, parse it with BeautifulSoup and lxml, send requests, handle JavaScript-heavy pages, crawl many pages politely, and store the results. This first lesson explains the idea, the pipeline every scraper follows, and the decisions you make before writing any code.

Why scrape at all?

Web pages are written for people. A product page shows a price in a large font next to a photo; a human reads it instantly. A program only sees text and tags. Scraping is the bridge: it converts "a page that looks right to a human" into "rows and columns a program can analyze".

Typical reasons people scrape:

  • Price monitoring. Track how the price of a product changes day by day.
  • Research and journalism. Collect public records, press releases or statistics that are published only as web pages.
  • Aggregation. Build a list of job posts, events or recipes from several sources.
  • Machine learning. Collect text or images to build a dataset.
  • Testing and auditing. Check your own website for broken links, missing titles or slow pages.

From page to dataset

A web page Written in HTML for humans.
A scraper Finds the pieces you want.
Structured data Rows and columns you can analyze.

Scraping turns visual pages into tables.

1 / 2

The four steps of every scraper

Whatever site you target and whichever library you use, a scraper does the same four things.

  1. Fetch. Send an HTTP request to a URL and receive a response that contains HTML (or JSON). Libraries: requests, httpx, a browser driver.
  2. Parse. Turn the raw text into a tree you can search. Libraries: BeautifulSoup, lxml, html.parser.
  3. Extract. Locate the elements you want with tag names, CSS selectors or XPath, and read their text or attributes.
  4. Store. Clean the values (strip whitespace, convert prices to numbers) and write them to a file or a database.
flowchart LR
    A["URL"] --> B["Fetch<br/>HTTP request"]
    B --> C["Parse<br/>HTML tree"]
    C --> D["Extract<br/>selectors"]
    D --> E["Clean<br/>and store"]
    E -.->|"next page"| A

The scraping loop: the last step often discovers the next URL, so the cycle repeats.

In this track the online TryIt editors cannot reach the internet, so the examples keep the HTML in a Python string. That is not a limitation of the technique: steps 2 to 4 work exactly the same on a string as on a page you downloaded. Only step 1 changes, and the lessons show the real network code as read-only snippets.

A first taste

Here is a tiny page held in a string, and a scraper that extracts every book title and price from it. Do not worry about the details yet; each piece gets its own lesson.

Python Your first scraper

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The script parsed the string into a tree, selected every li with class book, read the text of two child elements, converted the price to a number and printed a table. That is web scraping in about ten lines.

Scraping versus using an API

Before you scrape, ask whether the site already offers the data in a friendlier form.

Scraping HTML Using an API
Format HTML meant for people JSON or XML meant for programs
Stability Breaks when the layout changes Versioned and documented
Effort Parsing, cleaning Usually little parsing
Permission Often unclear Spelled out in the terms and keys
Speed Heavier pages Small responses

If a documented API gives you what you need, use it. It is faster, more reliable and clearly allowed. Scrape when no API exists, when the API omits fields visible on the page, or when it is too restricted or expensive for your use. Chapter 5 shows how to call APIs and how to find the hidden JSON endpoints many pages use behind the scenes.

Open your browser, right-click a price on any shop page and choose Inspect. The panel that appears shows the HTML your scraper will see. Learning to read it is the most useful scraping skill, and lesson HTML Structure for Scrapers covers it in detail.

Kinds of scrapers

Scrapers come in a few sizes:

  • A one-off script that downloads a single page and prints a few values. Ideal for learning.
  • A crawler that starts at one page and follows links through a whole site. Chapter 7 covers it.
  • A scheduled pipeline that runs every night, compares today's data with yesterday's and alerts you to changes.
  • A framework project such as Scrapy, which manages concurrency, retries and export for you.

You do not need a framework to begin. A script with requests and BeautifulSoup handles most small jobs and teaches the ideas that frameworks automate.

What can go wrong

Scraping is easy to start and tricky to keep running. Expect these problems, and the course will give you tools for each:

  • The layout changes, and your selectors stop matching. Write defensive code and add checks (Handling Messy HTML).
  • The data is not in the HTML. A page built by JavaScript delivers an empty shell; the data arrives later. See chapter 6.
  • The site blocks you. Too many requests too fast, or an obviously automated user agent, can get you rate-limited or banned. See chapter 7.
  • It may not be allowed. Terms of service, copyright and privacy law matter. See Legal and Ethical Basics.

Scraping a site gently, for public information, is usually fine. Scraping aggressively, ignoring a site's rules or collecting personal data is not. Read the legal lesson before you point a scraper at any real website.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which list shows the four steps of a scraper in the right order?

    You download the page (fetch), build a tree (parse), pick out the values (extract) and save them (store).
  2. 2. When should you prefer an official API over scraping?

    APIs return structured data, are documented and versioned, and make permission explicit, so they are the better choice whenever they cover your needs.
  3. 3. Why do the examples in this course keep HTML in a Python string?

    The sandbox cannot make network requests. Parsing and extraction behave identically on a string you already have; only the fetch step differs.
  4. 4. A page loads but your scraper finds none of the products you can see in the browser. What is a likely cause?

    Many modern pages deliver an empty shell and build the content with JavaScript. A plain HTTP request only sees the shell.

Summary

  • Web scraping uses a program to turn web pages into structured data.
  • Every scraper fetches, parses, extracts and stores; many loop back to fetch the next page.
  • Prefer a documented API when one exists; scrape when it does not.
  • Expect layout changes, JavaScript-built pages, blocking and legal questions, and plan for them.
  • In this course examples parse HTML strings because the editors have no network; the fetch code is shown read-only.

Next: Setting Up Your Scraping Toolkit