Skip to content
Tutorials Center
My learning

Topics

Lesson 13 of 38 ยท Target and Spider

Crawling and Content Discovery

Understand how Burp's crawler, the successor to the old Spider, maps a web app, how content discovery finds unlinked paths, and how to keep both polite and in scope.

Browsing by hand builds a good site map, but it only covers what you click. Two techniques fill the gaps: crawling, which follows links automatically, and content discovery, which guesses paths that no page links to.

You will often see this chapter's topic called the Spider. That was the old name for Burp's crawler tool. Since Burp 2020 the Spider has been replaced by the Crawler, which handles modern JavaScript-heavy sites and logins much better. The ideas below apply to both names.

The automated crawler and the Discover content tool are Burp Suite Professional features. In Community you map the site by browsing manually. The concepts still matter, and the stand-ins here let you practise the logic, so you will understand what the paid tools do.

How a crawler works

A crawler starts from one or more seed URLs, requests each one, extracts links, forms and scripts from the response, and queues the new URLs. It repeats until nothing new is found or a limit is reached.

flowchart TD
  S["Seed URL"] --> Q["Queue of URLs"]
  Q --> R["Request next URL"]
  R --> P["Parse links, forms, scripts"]
  P --> F{"In scope and new?"}
  F -->|"yes"| Q
  F -->|"no"| X["Discard"]
  R --> M["Add to site map"]

The loop that keeps expanding the site map until the queue is empty.

What makes a crawler hard is that a modern site is not just links. Pages build their content with JavaScript, forms need valid input to progress, and sessions can expire. Burp's crawler tackles this by driving an embedded browser (so scripts run), and by recognising that the same page may have several URLs and should not be crawled forever.

Starting a crawl (Professional)

From the Dashboard choose New scan, select Crawl only (not "Crawl and audit"), enter the URL, and then configure:

  • Scope: crawl only the in-scope URLs; this is the default and should stay.
  • Crawl optimization: how deep and how thorough the crawl is, trading time for coverage.
  • Login credentials: one or more test accounts for crawling authenticated areas. Use accounts created for testing, never real users.
  • Crawl limits: maximum time, request count and link depth.

Crawling sends real requests, including form submissions with plausible values, which can create records (orders, messages, accounts). In an authorised test, confirm with the owner whether crawling the production system is acceptable, and exclude destructive paths such as /logout and /delete-account.

Some paths exist but are not linked from anywhere: /admin, /backup, /old, /api/v2/internal. Content discovery (right-click a host in the site map and choose Engagement tools > Discover content) tries a wordlist of common names against the site and reports those that answer, as well as extensions and files such as .bak or .old.

Interpreting results needs care:

Response Meaning
200 The page exists and you can read it.
301/302 Redirect: often to a login or a canonical path.
401/403 The path exists but access is restricted: still a useful discovery.
404 Nothing there.
200 for everything The site returns a custom "not found" page with status 200; look at response length to separate real pages from fake ones.

Because discovery sends hundreds or thousands of requests, it is noisy. It can trigger alarms and, on fragile servers, slow them down. Use modest rate settings, run it only inside the agreed window, and tell the client if they asked to be informed.

Other discovery sources

  • robots.txt and sitemap.xml: public lists of paths the owner expects crawlers to see (or avoid, which is often interesting).
  • JavaScript files: they often contain API endpoints and hidden routes. Use Engagement tools > Find scripts and Find references.
  • HTML comments: developers leave notes and old paths in comments. Find comments lists them.
  • Source maps and API documentation (such as a published OpenAPI file) can describe whole APIs.

Try it: a crawler on a pretend site

This stand-in crawls an in-memory website (a dictionary of pages), so no real server is touched. It queues links, respects a scope rule, avoids loops and records what it found.

Python Stand-in: a crawler on an in-memory site

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The ad host was skipped (out of scope), /logout was excluded, and the back-links to /products did not cause an infinite loop because visited URLs are remembered.

Try it: content discovery against the same fake site

A second stand-in tries a wordlist against the in-memory site and classifies the answers, including a "soft 404" trap where one handler returns 200 for everything. No network is used.

Python Stand-in: content discovery with soft-404 detection

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Calibration is the key idea: ask for something that cannot exist, note what the "not found" response looks like, and subtract it. Burp's discovery tool does this fingerprinting for you.

Being a considerate crawler

  • Respect scope and exclusions; remove destructive URLs.
  • Use test accounts and a dedicated test environment where possible.
  • Limit speed so you do not overload the server.
  • Watch for forms that send email or create orders; a crawl can trigger real side effects.
  • Save results (the site map) and note the crawl's date and settings in your records.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What replaced Burp's old Spider tool?

    The Crawler is the modern successor, using an embedded browser to follow dynamic content.
  2. 2. What is content discovery for?

    Unlinked resources like /admin or /backup.zip are found by guessing names from a wordlist and examining the responses.
  3. 3. A server returns 200 for every nonexistent path with a short 'not found' page. What is the right approach?

    Soft 404s break status-code logic. Comparing against a known-missing baseline separates real pages from fake ones.
  4. 4. Why is crawling production with real forms sometimes risky?

    Crawlers fill in forms with plausible values. Exclude destructive paths and agree on rules with the owner.

Summary

  • Crawling follows links automatically; content discovery guesses unlinked paths from a wordlist.
  • Burp's Crawler replaced the Spider; the automated tools are Professional features, and Community relies on manual browsing.
  • Stay in scope, exclude destructive URLs, use test accounts, and limit speed.
  • Calibrate against a known-missing path to handle soft 404 responses.
  • Also mine robots.txt, sitemaps, scripts, comments and source maps for hidden paths.

Next: Mapping the attack surface