Skip to content
Tutorials Center
My learning

Topics

Lesson 40 of 43 ยท Projects

Project: Link Checker and Scraper

Build a polite Perl link checker: fetch a page with HTTP::Tiny, parse links with Mojo::DOM, honor robots.txt, rate-limit requests and test the pure parts.

A link checker visits a web page, collects every link on it and reports which ones are broken. It is a small program that touches many real-world topics: HTTP requests, HTML parsing, relative URLs, error handling, and good manners toward other people's servers. It is also a good project for learning how to structure a script so that the risky parts (the network) are separated from the parts you can test.

You will use modules, error handling, references and the testing tools from Testing with Test::More.

Perl and the network are not available on this site, so everything here is read-only. The pure helper functions and their test file were run on a real Perl 5.42 with the URI module and passed all 8 tests. The Mojo::DOM parts and the live HTTP requests were not run for this lesson, so there is no captured output for them. The interactive examples are Python stand-ins that show the same ideas on a page held in a string.

The plan

One page, many checks

Fetch Download the start page.
Extract Collect every href and make it absolute.
Ask permission Read robots.txt before visiting.
Check Send a HEAD request to each link.

Fetch, extract, filter, check, report.

1 / 2

Separate the risky parts from the pure parts

A function that takes a string and returns a string is easy to test: no network, no files, same answer every time. A function that downloads a page is not. So we put three pure helpers in a module, lib/LinkTools.pm, and keep the network code in the main script.

The helpers use the CPAN module URI, which is installed with many Perl distributions (otherwise cpanm URI).

PERL
package LinkTools;
use v5.36;
use Exporter 'import';
use URI;

our @EXPORT_OK = qw(resolve_link disallowed_prefixes is_allowed);

# Turn an href into an absolute http(s) URL without a fragment, or return undef.
sub resolve_link ($base, $href) {
    return undef unless defined $href && length $href;
    my $uri = URI->new_abs($href, $base);
    return undef unless $uri->scheme =~ /\Ahttps?\z/;
    $uri->fragment(undef);
    return $uri->canonical->as_string;
}

URI->new_abs does the hard work of combining a base URL with ../x?a=1 or /about. Links such as mailto:[email protected] or javascript:void(0) are not web pages, so the function returns undef for them. Removing the #fragment matters because /page#top and /page#end are the same download, and without this you would check one page many times.

Reading robots.txt

Most sites publish a file called /robots.txt that tells automated visitors which paths to avoid. A tiny parser is enough for the common cases. It keeps the Disallow lines of the group that applies to us (User-agent: * or a name that appears in our own user-agent).

PERL
sub disallowed_prefixes ($robots_txt, $agent) {
    my (@rules, $applies);
    for my $line (split /\r?\n/, $robots_txt) {
        $line =~ s/\s*#.*//;
        next unless $line =~ /\S/;
        if ($line =~ /\Auser-agent:\s*(.+?)\s*\z/i) {
            my $name = lc $1;
            $applies = ($name eq '*' || index(lc $agent, $name) >= 0);
        } elsif ($applies && $line =~ /\Adisallow:\s*(\S*)\s*\z/i) {
            push @rules, $1 if length $1;
        }
    }
    return @rules;
}

sub is_allowed ($path, @prefixes) {
    for my $prefix (@prefixes) {
        return 0 if index($path, $prefix) == 0;
    }
    return 1;
}

1;

Notice the small techniques: s/\s*#.*// strips comments, the /i flag makes the field names case-insensitive, and index($path, $prefix) == 0 is a fast "starts with" test that avoids building a regex from user data. This parser ignores Allow lines, wildcards and Crawl-delay, so for serious crawling use a maintained module such as WWW::RobotRules.

Testing the pure parts

These tests run in milliseconds and need no network:

PERL
use v5.36;
use Test::More;
use lib 'lib';
use LinkTools qw(resolve_link disallowed_prefixes is_allowed);

my $base = 'https://example.com/docs/index.html';

is resolve_link($base, 'intro.html'),         'https://example.com/docs/intro.html', 'relative link';
is resolve_link($base, '/about#team'),        'https://example.com/about',           'fragment removed';
is resolve_link($base, '../x?a=1'),           'https://example.com/x?a=1',           'parent directory';
is resolve_link($base, 'mailto:[email protected]'),    undef,                                 'mailto skipped';
is resolve_link($base, 'javascript:void(0)'), undef,                                 'javascript skipped';

my $robots = <<'TXT';
# example
User-agent: *
Disallow: /private/
Disallow: /tmp   # scratch
User-agent: BadBot
Disallow: /
TXT

my @rules = disallowed_prefixes($robots, 'LinkCheck/1.0');
is_deeply \@rules, ['/private/', '/tmp'], 'rules for the star group';
ok  is_allowed('/docs/a.html', @rules),  'normal path allowed';
ok !is_allowed('/private/x',  @rules),   'private path blocked';

done_testing;

Running perl t/links.t printed ok 1 to ok 8, then 1..8. The BadBot group is correctly ignored because our agent name does not contain it.

Fetching with HTTP::Tiny

HTTP::Tiny has been part of Perl itself since 5.14, so it needs no installation. Create one client object with a clear identity and a timeout, and reuse it. HTTPS needs the IO::Socket::SSL and Net::SSLeay modules to be installed.

PERL
my $http = HTTP::Tiny->new(
    agent        => 'LinkCheck/1.0 (+mailto:[email protected])',
    timeout      => 10,
    max_redirect => 5,
);

my $res = $http->get($start_url);
die "Cannot fetch $start_url: $res->{status} $res->{reason}\n" unless $res->{success};

HTTP::Tiny never throws an exception on a network problem. It returns a hash reference with success, status, reason, headers and content. When the connection itself fails (a refused connection, a timeout), the status is the made-up code 599 and content holds the error text. For example, a request to a closed local port returned status 599 with the reason Internal Exception and the message Could not connect to '127.0.0.1:9': Connection refused. Always test success, and treat 599 as "could not reach the server" rather than "the page is broken".

Regular expressions are a poor tool for HTML, because markup can nest, break across lines and contain odd quoting. Use a parser. Mojo::DOM (from the Mojolicious distribution, cpanm Mojolicious) understands CSS selectors, so the selector a[href] means "every anchor that has an href".

PERL
use Mojo::DOM;

my $dom   = Mojo::DOM->new($res->{content});
my $title = $dom->at('title') ? $dom->at('title')->text : '(no title)';

my %seen;
my @links = grep { defined && !$seen{$_}++ }
            map  { resolve_link($start_url, $_) }
            $dom->find('a[href]')->map(attr => 'href')->each;

Read it from the bottom up: find returns every matching element, map(attr => 'href') turns each element into its href text, resolve_link makes it absolute (or undef), and grep drops the undefs and any URL already seen. The hash %seen is the "set" idiom from Hash Patterns.

A HEAD request asks for the headers only, which is cheaper than downloading the page. Some servers do not support it and answer 405 or 501, so fall back to GET in that case. The pause between requests keeps the program from hammering one server.

PERL
use Time::HiRes qw(sleep time);

my %last_hit;     # host => time of the last request
my $delay = 1.0;  # seconds between requests to the same host

sub polite_wait ($host) {
    my $wait = ($last_hit{$host} // 0) + $delay - time;
    sleep $wait if $wait > 0;
    $last_hit{$host} = time;
}

sub check ($http, $url) {
    my $host = URI->new($url)->host;
    polite_wait($host);
    my $res = $http->head($url);
    $res = $http->get($url) if $res->{status} =~ /\A(?:405|501)\z/;
    return $res;
}

for my $url (@links) {
    my $res = check($http, $url);
    my $label = $res->{success}         ? 'OK '
              : $res->{status} == 599   ? 'ERR'
              :                           'BAD';
    printf "%s %3d %s\n", $label, $res->{status}, $url;
}

time and sleep from Time::HiRes accept fractions of a second, unlike the built-in sleep, which rounds to whole seconds. The delay is kept per host, so links to ten different sites are not slowed down, but links to the same site are spaced out.

  • Read the site's terms. Some forbid automated access, and robots.txt is a request about crawling, not a permission slip for everything you could do with the data.
  • Identify your program in agent with a way to contact you.
  • Do not crawl in parallel against one host. One request at a time, with a delay, is a good default.
  • Cache what you download while developing. Do not re-fetch the same page on every test run.
  • Scraping your own site, or a site that offers an API, is the safe case. If a site has an API, use that instead.

A runnable stand-in

Here is the extraction and filtering step in Python, working on a page held in a string. Python's html.parser and urljoin play the roles of Mojo::DOM and URI->new_abs, so you can see exactly how relative links, fragments and mailto: links are handled.

Python Extract and resolve links (Python stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that intro.html and intro.html#part2 collapse into one entry, and the two non-web links are reported as skipped.

The second stand-in shows the robots.txt idea with the standard library parser, again on text held in a string, so no network is needed.

Python Honor robots.txt (Python stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The same site gives different answers to different agents, just as the User-agent groups in the Perl parser do.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why is a parser such as Mojo::DOM better than a regular expression for finding links in HTML?

    Markup has too many valid forms for a simple pattern to cover. A parser understands the structure, and CSS selectors like a[href] describe what you want.
  2. 2. What does status 599 from HTTP::Tiny mean?

    599 is not a real HTTP status. HTTP::Tiny uses it for internal failures and puts the error message in the content.
  3. 3. Why remove the #fragment from each link before checking it?

    Browsers keep the fragment to themselves, so removing it avoids checking the same page repeatedly.
  4. 4. Why put resolve_link and the robots.txt helpers in a module instead of the main script?

    Pure helpers give the same output for the same input, so tests are fast and reliable. The network code stays small and separate.
  5. 5. What is a polite default when checking many links on the same host?

    A delay protects the server from overload, and a clear User-Agent lets the owner contact you.

Summary

  • Keep network code thin and put pure helpers (URL resolving, robots rules) in a module you can test.
  • URI->new_abs resolves relative links; drop fragments and non-web schemes.
  • HTTP::Tiny is core Perl, returns a result hash instead of dying, and uses status 599 for connection failures.
  • Parse HTML with a real parser such as Mojo::DOM and CSS selectors.
  • Use HEAD with a GET fallback, a per-host delay, a clear User-Agent and robots.txt.
  • Respect site terms; prefer an official API when one exists.

Next: Practice Exercises with Solutions.