Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 39 ยท Selectors and Data Extraction

Extracting Links, Images and Metadata

Extract links, images and srcset, page titles, meta and Open Graph tags, canonical URLs and JSON-LD structured data from HTML with BeautifulSoup and the json module.

Beyond the visible content, every page carries a layer of machine-readable information: link targets, image sources, titles, descriptions, social-sharing tags and structured data. This layer is often easier to scrape than the visible text, and it is the foundation of crawlers (which follow links) and of SEO audits (which check titles and descriptions). This lesson shows how to extract it all.

Every a tag with an href is a link. A crawler cares about where it goes and what kind of link it is.

Python Collect and classify links

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notes for crawlers: skip mailto:, tel:, javascript: and fragment-only links; resolve relative links with urljoin; compare the host to separate internal from external links; and honor rel="nofollow" if you are building something search-like. The download attribute marks a file link, and file extensions such as .pdf tell you the type without a request.

Link text is also data: a.get_text(strip=True) is the anchor text, used in SEO analysis.

Images

img tags use src, but modern pages complicate it:

  • srcset lists several sizes: small.jpg 480w, large.jpg 1200w.
  • <picture> with <source> offers alternative formats.
  • Lazy loading often puts the real URL in data-src or data-lazy-src, leaving src as a tiny placeholder.
  • alt text describes the image and is useful for accessibility checks and indexing.
Python Find the real image URL

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Downloading an image is then one GET to that URL, saving response.content as bytes. Use the file extension or the Content-Type header to name the file, and respect size and copyright limits.

Title, meta tags and the canonical URL

The <head> holds page-level information:

HTML
<title>Desk Lamp | Corner Shop</title>
<meta name="description" content="A bright LED lamp.">
<meta name="robots" content="index,follow">
<link rel="canonical" href="https://shop.example.com/p/1">
<link rel="alternate" hreflang="de" href="https://shop.example.com/de/p/1">
<meta property="og:title" content="Desk Lamp">
<meta property="og:image" content="https://shop.example.com/i/lamp.jpg">
<meta name="twitter:card" content="summary_large_image">
Python Read the head

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The pattern meta(name="description") passes arbitrary attributes through attrs=, because name is also the argument that names the tag in find. This head data supports a simple SEO audit: check that every page has one title (under about 60 characters), a description (under about 160), a canonical link and no accidental noindex.

Structured data: JSON-LD

Search engines ask sites to embed structured data, and the most common format is JSON-LD inside a script tag. For a scraper that is a gift: the product name, price, rating, availability and dates are already clean JSON.

Python Read JSON-LD

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real-world JSON-LD is messy: a block can be a list, @type can be a list, offers can be a list of offers, and values may be strings or numbers. Write defensive code and wrap json.loads in try/except. Related formats you may meet are microdata (itemscope, itemprop attributes), RDFa, and the window.__NEXT_DATA__ or __NUXT__ JSON that frameworks embed; the latter often contains the entire page data and is worth looking for before parsing the HTML at all.

When a site offers JSON-LD or embedded JSON, use it. It changes far less often than the visual markup, so your scraper stays alive longer.

Sitemaps and feeds

Two more machine-readable sources save crawling effort. sitemap.xml lists the URLs a site wants indexed; RSS and Atom feeds list recent items. Both are XML, which BeautifulSoup parses with the "xml" parser (needs lxml):

Python
soup = BeautifulSoup(sitemap_text, "xml")
urls = [loc.get_text(strip=True) for loc in soup.find_all("loc")]

Reading a sitemap is more polite and efficient than crawling link by link, and its lastmod dates tell you which pages changed.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A lazy-loaded image has a tiny src placeholder. Where is the real URL likely to be?

    Lazy loading commonly stores the real address in data-src (or similar) or lists it in srcset.
  2. 2. Which links should a crawler skip when collecting pages to visit?

    Those links are not web pages to request, so they should be filtered out.
  3. 3. Why is JSON-LD attractive to scrapers?

    Structured data is meant for machines and tends to stay stable, though you must still handle malformed blocks.
  4. 4. How do you read the content of an Open Graph tag such as og:title?

    Open Graph tags are meta elements with a property attribute and the value stored in the content attribute.

Summary

  • Collect links with find_all("a", href=True), resolve them with urljoin, and filter mailto:, tel:, javascript: and fragments.
  • Find the real image URL in srcset, data-src or src.
  • Read title, meta description, robots, canonical, hreflang and Open Graph tags from the head.
  • Parse JSON-LD script blocks with json.loads defensively; look for other embedded JSON too.
  • Sitemaps and feeds are tidy XML sources of URLs.

Next: Handling Messy HTML