Skip to content
Tutorials Center
My learning

Topics

Lesson 137 of 147 ยท Projects

Project: Concurrent Web Crawler

Build a web crawler in Go in four stages - breadth-first crawling of a simulated site, bounded concurrency with a mutex-protected visited set, context timeouts and cancellation, and a full report.

A web crawler starts at one page, finds the links on it, visits those pages, finds their links, and carries on until it has seen the whole site or hit a limit. Search engines are built on crawlers, but so are link checkers, site-mapping tools and archive builders. The task is small to describe and surprisingly rich to build well, because a crawler is almost entirely waiting: most of its time goes to the network, not to the CPU. That makes it the textbook case for Go's concurrency, and also a place where mistakes are expensive. Without limits, a crawler overloads servers, follows links forever, and hangs on the first page that never answers.

In this project you build a crawler in four stages, and each stage is a complete program. In stage 1 you write the sequential version: fetch a page, extract and resolve its links, and walk the site breadth-first with a visited set and a depth limit. In stage 2 you make it concurrent with bounded parallelism and a mutex-protected visited set. In stage 3 you add context: a timeout for every request, a deadline for the whole crawl, and handling of pages that fail in nasty ways. In stage 4 you add a collector goroutine that aggregates everything into a report with broken-link referrers and a site outline.

You'll use the HTTP client, net/url, regexp, goroutines, sync, channels and context.

A crawler must never be pointed at servers you do not own, so this project crawls a simulated web. The program builds a small website in memory (a map from path to the links on that page) and serves it with net/http/httptest.NewServer on a random loopback port. It is a real HTTP server, so the code is exactly what you would write for the real thing, but nothing leaves your machine. The port differs on every run, so the output only ever shows paths.

The plan

The site has 15 pages, including pages that link to each other in loops, relative links such as ../blog/post-1, a mailto: link, links with #fragments, links to other websites, two links to pages that do not exist (404), and a chain of pages that is four levels deep.

Stage What you build Main Go ideas
1 A sequential breadth-first crawler net/url resolving, regexp, queue and visited set, io.LimitReader
2 Concurrent fetching with a worker limit Semaphore channel, sync.Mutex, WaitGroup.Go, levels as barriers
3 Timeouts, cancellation, failing pages context.WithTimeout, NewRequestWithContext, errors.Is, mime
4 A collector goroutine and a full report Fan-in over a channel, single-owner data, deterministic output

Stage 1: the sequential crawler

A crawler repeats one small step, "fetch a page and find its links", so write that step first and test it alone.

Fetching. fetch makes a GET request with an http.Client that has a timeout. This is not optional: Go's default client, http.DefaultClient, has no timeout, and one unresponsive server can block a program forever. The body is read through io.LimitReader(resp.Body, 1<<20), so a huge or endless response can cost at most 1 MiB of memory. defer resp.Body.Close() is also mandatory, or the connection cannot be reused (see the HTTP client).

Extracting links. HTML in real life is messy, and the right tool is a proper HTML parser (the golang.org/x/net/html package is the usual choice, but it is not in the standard library, which this course sticks to). The pages of our simulated site are generated by our own code, so a regular expression for <a href="..."> is enough, and it keeps the example short. Do not use regular expressions for arbitrary HTML in production.

The raw href values need cleaning before they can be followed:

  1. Resolve them against the page's own URL with base.ResolveReference(ref). This turns /team, post-2 and ../blog/post-1 into full URLs, following the same rules as a browser.
  2. Drop the fragment, since /about#history is the same page as /about.
  3. Skip anything that is not http or https, such as mailto: links.
  4. Skip other hosts. A crawler that follows every external link would try to download the whole internet. Staying on the same host is the simplest policy.
  5. De-duplicate within the page.

Walking the site. Breadth-first search uses a queue: take the first item, process it, append its new links to the end. A visited map prevents revisiting, which is essential because sites are full of cycles (every page links back to the home page). Marking a URL as visited at the moment it is queued, not when it is fetched, keeps duplicates out of the queue. The depth of a page is 1 + the depth of the page where it was first found, and the crawler does not follow links from pages at the maximum depth. Breadth-first order guarantees that the first time a page is found is also its shortest path from the start.

The program crawls the same site twice, with depth limits of 1 and 3.

Go Stage 1: a sequential breadth-first crawler

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Look at the output. With depth 1 the crawler fetches the start page and the pages it links to, but does not read their links. With depth 3 it reaches 14 pages. The links= column counts distinct same-host links and skipped= counts the ignored ones: the home page skips its mailto: link, /about skips the external partner link. The two 404s at depth 3 (/blog/draft and /products/missing) are found, because the crawler follows links to depth 3, but they have no links of their own. The /deep chain is cut off after /deep/1, since /deep/2 would be at depth 4.

Design points:

  • The crawler "sees" /team's link ../blog/post-1 as /blog/post-1. Without ResolveReference, relative links would be the most common source of bugs.
  • visited holds url.String(), the complete URL. Two different spellings of the same page (/a and /a/, upper and lower case host, ?utm_source=...) are still different keys. Real crawlers normalize URLs more aggressively.
  • A 404 is a result, not an error. Result.Status records it, and the crawl goes on.

Stage 2: concurrent fetching with a limit

Almost all of the 14 pages' time is spent waiting for a server. While one request waits, the program could be sending others. Concurrency lets those waits overlap, and in Go it takes very little code: one goroutine per page.

But there are two dangers. Unlimited concurrency would open a connection per discovered link, which on a big site means thousands of simultaneous requests: your machine runs out of file descriptors, and the target server may ban you. And shared state (the visited set, the result list) needs protection from data races.

The design of stage 2:

  • Level by level. The crawler processes one depth level at a time: all pages at depth d are fetched concurrently, then it waits (wg.Wait()), then moves on to depth d+1. This barrier has a valuable consequence: when a worker finds a link, every page of the previous levels has already been handled. A link first discovered at level d+1 therefore really is at distance d+1, so the depth numbers and the depth limit behave exactly like in the sequential breadth-first search. (A fully free-running crawler, where goroutines spawn goroutines, can find a page first through a long path and then wrongly cut it off at the depth limit.)
  • A counting semaphore limits requests in flight. It is a buffered channel with room for Workers values: sending a value takes a slot, receiving gives it back, and when all slots are taken the next sender blocks. A goroutine is cheap, so there is one per URL, but at most Workers of them are talking to the network at any moment. See concurrency patterns.
  • One mutex guards the shared data: the visited map, the next list of URLs for the following level, and the results slice. Each critical section is a few lines and does no I/O. The mutex is never held while a request is running.
  • WaitGroup.Go (Go 1.25) starts the goroutines and does the Add/Done bookkeeping.
  • Results are sorted at the end, by depth and then by path, so the output no longer depends on which goroutine finished first.

The program crawls the site with 1, 2, 4 and 8 workers. The simulated pages each take 25 ms to answer, as a real server would. It checks that every run found exactly the same pages with the same depths and statuses, and that eight workers beat one worker by at least a factor of two.

Go Stage 2: bounded concurrency with a mutex

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Every worker count gives the same pages: concurrency changed how fast the crawl runs, not what it finds. That equality is the most important check in the program. For a speedup, 14 pages at 25 ms each take about 350 ms one at a time, while eight workers need roughly one 25 ms wait per level. The speedup comes from overlapping waiting, not from using more processors, so it appears even on this one-CPU sandbox. (The timing check prints true with a wide safety margin, but never rely on exact timings in tests.)

Some details of this design are worth a closer look:

  • The semaphore is acquired inside the goroutine. All goroutines for a level are created at once, and the semaphore makes most of them wait for a slot. Creating them is cheap; what must be limited is the work.
  • The mutex protects enqueue and record only. fetch, which does the slow network work, runs without any lock. Holding a lock across I/O would serialize the crawler again.
  • The unlocked read of c.next after wg.Wait() is safe, because Wait returns only after every goroutine has called Done, and that establishes a happens-before relationship. See data races.
  • Closures capture u and depth per iteration (Go 1.22), so each goroutine sees its own page. Before Go 1.22 you had to copy the loop variable by hand.

The level barrier has a cost: one slow page delays the start of the next level for everyone. A crawler without barriers would keep all workers busy at all times, but needs more care about depth and termination (the usual design is a coordinator goroutine owning the queue and a count of pages in flight). The level-by-level structure is simpler and its results are reproducible.

Stage 3: timeouts, cancellation and failing pages

The real internet is not as friendly as our 14-page site. Servers answer slowly, answer with errors, return PDFs where you expect HTML, or never answer at all. A crawler must survive all of these. Stage 3 adds three kinds of protection, all built on context:

  1. A per-request timeout. fetch derives a child context with context.WithTimeout(ctx, c.Timeout) and builds the request with http.NewRequestWithContext. When the time is up, the request is aborted and Client.Do returns an error that wraps context.DeadlineExceeded.
  2. A deadline for the entire crawl, set by the caller. When it expires, in-flight requests are aborted, the level loop sees ctx.Err() != nil and stops, and goroutines still waiting for a semaphore slot record a "canceled" result without starting. The crawler never blocks forever on a slot, because acquiring it is a select between the semaphore and ctx.Done().
  3. Content checks. The server's Content-Type is parsed with mime.ParseMediaType. Only text/html is read and scanned for links; a PDF is recorded and left alone.

Errors need translating for humans. A *url.Error message contains the full URL, with the random port. The helper describe uses errors.Is(err, context.DeadlineExceeded) to say timeout and errors.As to dig out the inner error otherwise. See errors.

The site now has four misbehaving pages linked from the home page: /slow (answers after 40 ms and is perfectly fine), /hang (never answers until the client gives up), /broken (HTTP 500) and /files/report.pdf (a document). The first run gives each request 200 ms and no overall limit, so /hang becomes a timeout result and the crawl carries on. The second run has no per-request limit but a 300 ms deadline for the whole crawl: the hanging request is aborted when the deadline passes, and the pages below /slow and the other level-1 pages are never fetched.

Go Stage 3: request timeouts and a crawl deadline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The first run recorded 16 pages. Its problem list shows the 500, the PDF that was recorded but not parsed, the timeout (status 0 means "no response"), and the two 404s. Everything else, including the deep pages below /slow, was crawled normally. The second run recorded only 8 pages: the home page and its seven neighbors. When the deadline passed, the /hang request was aborted (reported as timeout, because the context ended by deadline), the loop saw that the context was done, and nothing from level 2 was started. Finally, ctx.Err() confirms why the crawl stopped.

Always defer cancel() after context.WithTimeout, even if the timer fires by itself: calling cancel releases the timer and the resources of the context right away. go vet reports a forgotten cancel function. The same advice appears in the context lesson.

Two more details:

  • The server handler for /hang waits for <-r.Context().Done(). When the client aborts the request, the server notices the closed connection and its request context is canceled, so the handler returns. This is also how real handlers should stop expensive work for clients that left.
  • A Client.Timeout still exists on the client (10 seconds here) as a last line of defense, but the contexts are more precise: they can differ per request and per crawl.

Stage 4: collecting the results into a report

Up to now each worker appended to a slice under a mutex. That works, but it spreads knowledge about the result around the program. Stage 4 uses the other classic style: workers send finished pages on a channel to one collector goroutine, and only the collector ever touches the report data. There is no mutex around the report, because only one goroutine can access it. This is the Go proverb in action: do not communicate by sharing memory; share memory by communicating.

The code is organized in layers:

  • Options groups the settings: depth, workers and timeout.
  • Page holds everything learned about one URL: status, title, size in bytes, same-site links, and the hosts of external links.
  • Crawler.Run starts collect in a goroutine, runs the level loop, sends every Page into pagesCh, closes the channel when the crawl is over and waits for the finished *Report on a second channel.
  • collect builds the report: the page list, the referrers of every link (which pages point to each URL) and a count of external hosts. Since pages arrive in random order, it sorts at the end, and it sorts and de-duplicates each referrer list.
  • Report.Render produces the text: summary, pages per depth, external hosts, broken links with their referrers, and an outline. The outline hangs each page under the first page (in sorted order) from the previous level that links to it, so the tree is the same whatever the arrival order.

The visited set and next list still need the mutex, because workers add to them concurrently while fetching. Not everything can be moved into one owner: choose per piece of data.

The final check crawls the site with 4 workers and prints the report. Then it crawls with 1, 2 and 16 workers and compares the rendered text with the first one. If any ordering or counting depended on timing, this comparison would fail from time to time.

Go Stage 4: collector goroutine and full report

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Read the report from top to bottom. The summary says 15 URLs were fetched: 13 fine and 2 broken, in 5 levels (depth 0 to 4), with the page counts per depth underneath. Three external links were found on the fetched pages: two to partner.example and one to shop.example. Each broken link names the page that contains it, which is the information you need to fix the site. In the outline, /deep/2 appears at depth 4 and has no children, because the crawler does not follow links from pages at the depth limit. The last line confirms the report is the same for every worker count.

Being a good citizen

A crawler on the real internet has obligations that our simulated site does not test:

  • Respect robots.txt, which tells crawlers which paths to avoid. Fetch it first and obey it.
  • Limit the rate per host, not just the total concurrency. A few requests per second per host is polite; use time.Ticker or a token bucket (see concurrency patterns).
  • Identify yourself with a descriptive User-Agent header and a contact address.
  • Follow redirects deliberately. http.Client follows up to ten by default. Record the final URL, and treat it as visited as well, or the same page will be fetched twice.
  • Cap everything: depth, number of pages, size per response, total time. We have used depth, size and time; add a page budget too.
  • Normalize URLs (lowercase the host, remove default ports and tracking parameters, sort query parameters) before checking the visited set.
  • Retry carefully: only for temporary failures (timeouts, 502, 503), with exponential backoff, and never for 4xx.

Ideas for extending the crawler

  • Add a page budget (MaxPages) and stop the crawl by calling cancel() when it is reached.
  • Replace the regular expression with the golang.org/x/net/html tokenizer, which understands <base href>, rel="nofollow" and unquoted attributes.
  • Implement the free-running design: a coordinator goroutine that owns the queue and counts in-flight pages, so there are no level barriers.
  • Verify external links too, with HEAD requests, and report the dead ones.
  • Write the report as JSON, or the outline as an XML sitemap (see CSV, XML and other formats).
  • Honor robots.txt and Retry-After headers, and add a per-host rate limiter.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does the crawler build its http.Client with a Timeout instead of using http.DefaultClient?

    The default client waits as long as the server does. Always set a timeout (and prefer per-request contexts) when talking to servers you do not control.
  2. 2. In stage 2, what does the buffered channel slots := make(chan struct{}, c.Workers) implement?

    Sending takes a slot and receiving frees it. Once Workers slots are in use, the next send blocks until some goroutine finishes and receives.
  3. 3. Why does stage 2 wait with wg.Wait() between depth levels instead of letting goroutines spawn goroutines freely?

    With a barrier, all shorter paths have been explored before a deeper level starts. A free-running crawler can find a page first through a long path and then stop at the depth limit too early.
  4. 4. How do you tell that a request failed because its context deadline passed?

    Client.Do wraps the context error in a *url.Error. errors.Is follows the chain and finds DeadlineExceeded; a plain == comparison would fail.
  5. 5. In stage 4 only the collector goroutine touches the Report, so there is no mutex around it. Which rule makes that safe?

    Ownership by a single goroutine removes the race. Workers only send values; the collector alone reads and writes the report data.

Summary

  • Fetch with a client that has a timeout, limit the bytes you read, and always close the body. Resolve links with URL.ResolveReference, drop fragments, and filter by scheme and host.
  • Breadth-first search needs a queue and a visited set; mark URLs visited when they are queued so that cycles cannot grow the queue. The depth of a page is its shortest distance from the start.
  • To crawl concurrently, bound the parallelism with a buffered-channel semaphore, protect shared maps and slices with a mutex held only for short, I/O-free sections, and wait for each level with a WaitGroup so results stay deterministic.
  • Use context for every network call: a timeout per request, a deadline or cancel for the whole crawl, and select on ctx.Done() wherever you might block. Translate errors with errors.Is and errors.As.
  • Check content types before parsing, and treat HTTP errors as results to record, not reasons to crash.
  • For aggregation, let workers send values to one collector goroutine that owns the report. Sort in the end so that the output does not depend on timing, and test by comparing runs with different worker counts.

Next: Project: Key-Value Store with TCP Protocol