Skip to content
Tutorials Center
My learning

Topics

Lesson 13 of 53 ยท Reconnaissance and Scanning

Content Discovery with Gobuster and Nikto

Learn how directory, DNS and virtual host discovery tools work, how wordlists and status codes guide results, what Nikto checks, and how defenders reduce and detect this noise.

Web servers often hold more than the pages linked from the home page: an old /backup folder, an admin panel, a staging site on the same address. Content discovery is the practice of guessing paths and hostnames from a wordlist and watching how the server responds. It is not clever; it is patient. Its value is finding things that were never meant to be public but were never protected either.

Content discovery sends hundreds or thousands of requests. On a real site that looks like an attack, can overload small servers and can trip rate limits and alarms. Run it only on targets in your written scope, ideally your own lab applications such as DVWA. The runnable examples here use a fake server in memory.

Three discovery modes

Tools such as Gobuster (and similar programs) are essentially fast, polite loops around one idea: take a word, build a request, record the result. They differ in what the word becomes.

  • Directory (dir) mode appends words to a base URL: /admin, /backup, /uploads. Extensions can be tried too (.php, .bak, .old) because forgotten copies of source files are a classic find.
  • DNS mode prepends words to a domain: dev.example.test, vpn.example.test. Each name either resolves or does not.
  • Virtual host (vhost) mode keeps the IP address but changes the HTTP Host header: one web server may serve www, staging and internal sites that only differ by that header. A name that does not appear in DNS can still answer.

Read-only examples of what these look like:

TEXT
gobuster dir   -u http://lab.test -w common.txt -x php,bak
gobuster dns   -d example.test -w subdomains.txt
gobuster vhost -u http://192.0.2.80 -w subdomains.txt

Wordlists matter more than speed

A discovery run finds only what is in the list. A short list of common names finds the obvious; a technology-specific list (names typical for a content management system) finds framework leftovers; a custom list built from the target's own vocabulary (product names, project code names, words from the site) finds what generic lists miss. Collections of lists are packaged for security work and installed on Kali. Bigger is not always better: a million-word list against a fragile server is both slow and impolite.

Speed settings (threads, delays) are part of your rules of engagement. Agree them with the client: a production site on a small server may only tolerate a few requests per second.

Reading status codes

The status code is the signal. But "found" is not just 200 OK.

Code Usual meaning for discovery
200 Exists and is readable
301 / 302 Exists but redirects (often a directory without a trailing slash, or a login page)
401 Exists and requires authentication, a promising target
403 Exists but access is forbidden, still confirms it is there
404 Does not exist
429 You are being rate limited, slow down
5xx Server error, possibly triggered by your request

Some servers return 200 for everything and show a friendly "not found" page. These soft 404s make naive tools report thousands of false hits; good tools compare response sizes or let you filter by length. Always check a few results by hand.

A fake server and a simple discovery loop

The stand-in below defines a server as a lookup table, a wordlist and a loop. It classifies results and removes soft-404 noise by comparing the response length to a known "not found" page.

Node.js Discover paths on an in-memory server

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The interesting finds are the .bak file (source code and settings in a copy that the server happily serves as text) and the 401 and 403 paths that confirm hidden areas. In a real engagement each finding goes into the notes with the request that proved it.

Nikto and other web scanners

Nikto is a web server scanner. Rather than guessing from a generic wordlist, it checks a large database of known problems: dangerous default files, outdated server software, risky HTTP methods, missing security headers, exposed administration pages. It is thorough but noisy, so it is easy to detect and rarely used when stealth matters. It also produces false positives, so every item needs manual verification.

TEXT
nikto -h http://lab.test -o nikto-report.txt

Treat scanner output as leads. A line such as "directory indexing found" or "outdated server version" is a question to answer, not a finished finding. Pair it with a manual look and with vulnerability scanning results, then confirm what is real before it reaches the report.

Limits of discovery

  • It finds only what is guessable; unlinked paths with random names stay hidden.
  • Authentication, rate limits and web application firewalls change the picture.
  • A 403 can mean "forbidden for everyone" or "forbidden from your IP".
  • Results depend on the Host header, the user agent and sometimes the session.
  • Heavy scanning can damage availability. Respect the agreed speed.

Defending against content discovery

The best defense is not hiding names but not leaving things behind.

  • Keep backups, source copies and configuration files out of web roots; deploy from a build, not from a working directory.
  • Return identical responses for missing and forbidden items where it makes sense, and avoid soft 404s that confuse your own monitoring.
  • Disable directory listing and remove default files and sample applications (see server misconfiguration).
  • Protect every administrative path with authentication and, where possible, network restrictions, rather than relying on an unguessable name.
  • Rate-limit and log: many 404 responses from one source is a clear signal.

The second stand-in implements that signal, counting 404s per source in a window.

Node.js Detect a discovery run in access logs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Dirb: how it works and usage

Dirb is a classic content scanner that requests paths from a wordlist against a web server and reports the response codes. It works by launching a dictionary of likely names (admin, backup, uploads, old), optionally adding extensions, and recording which return 200, 301 or 403 rather than 404.

Common usage ideas, always on systems you are authorized to test:

  • run with a target URL and a wordlist; the default list is small, larger lists find more;
  • add file extensions to look for backup copies;
  • send a custom user agent or cookie to reach authenticated areas;
  • speed delaying (a delay option such as -z) slows requests so a fragile server is not overloaded and rate-limiters are not triggered;
  • non-recursive mode (-r) stops it from descending into every directory it finds, saving time.

Defenders can spot such scans as bursts of 404 responses from one address, and prevent exposure by never leaving backups or admin tools in the web root, returning the same response for missing and forbidden pages, and rate-limiting.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does virtual host discovery change in the request?

    A single server can host several sites distinguished by the Host header. Trying names from a wordlist reveals ones that DNS does not list.
  2. 2. A path returns 403 Forbidden. What does that tell a tester?

    403 usually confirms the resource exists. Whether it is exposed depends on the access rules, not the code.
  3. 3. Why are soft 404 pages a problem for discovery tools?

    When missing pages return 200, status codes alone cannot separate real from fake. Comparing response size or content helps.
  4. 4. Which defense removes the main prize of directory discovery?

    Discovery finds leftovers. If nothing sensitive is deployed, a successful guess finds nothing useful.

Summary

  • Content discovery guesses paths, subdomains and virtual hosts from a wordlist and interprets the responses.
  • Gobuster-style tools offer directory, DNS and vhost modes; the wordlist and request speed shape the result and must fit the rules of engagement.
  • Status codes 200, 301/302, 401 and 403 usually mean "exists"; soft 404s need calibration to filter.
  • Nikto checks a database of known web server issues; verify every item manually.
  • Defenders remove leftovers from web roots, authenticate admin areas, rate-limit and alert on bursts of 404s.

Next: Packet Analysis with Wireshark