Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 35 ยท Quality and Reliability

Security, Ethics and Scraping Limits

Keep Selenium tests safe and responsible: handle secrets and test accounts, respect robots.txt and terms of service, throttle requests, and know when not to scrape.

A browser that can click anything can also break things: delete a real customer's order, lock a production account, leak a password into a public log, or hammer someone else's server. Selenium gives you a lot of power with no built-in guard rails, so the guard rails have to come from you and your team. This lesson covers the habits that keep a test suite safe, and the limits you should respect when automation touches websites you do not own.

It has two halves. The first is about testing your own application: secrets, test accounts and environments. The second is about other people's sites: robots.txt, terms of service, rate limits, and why Selenium is a poor tool for large-scale scraping.

This lesson is general engineering guidance, not legal advice. Rules about automated access differ between countries and sites. When money, personal data or a contract is involved, ask someone qualified.

Two kinds of responsibility

Your own app Protect secrets, data and environments.
Other sites Respect their rules and their servers.

Same tool, very different obligations.

1 / 2

The code in this lesson is read-only or runs as stand-ins. The site cannot launch a browser, so the three runnable examples model the logic (parsing robots.txt, loading secrets, scheduling polite requests) in plain JavaScript.

Secrets in tests

Tests need credentials: a login, an API token, maybe a payment sandbox key. The classic mistake is writing them straight into the test:

Python
driver.find_element(By.ID, "password").send_keys("Hunter2!")   # never do this

That string is now in version control forever, in every clone, and often in CI logs. Instead:

  1. Read secrets from the environment. Locally from a .env file that is listed in .gitignore; in CI from the platform's secret store (see Secrets, Variables and Environments).
  2. Fail fast when one is missing, with a message that names the variable, not the value.
  3. Mask values in logs and reports. Your page objects may log every send_keys; do not log the text for password fields.
  4. Rotate and scope. Use credentials that can only touch the test environment, and change them if they ever leak.

Here is a stand-in that does exactly that: it refuses to start with missing variables, masks the password in a log line, and refuses to run against a host that is not on an allow-list.

JS Load secrets safely and guard the target

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The first call fails with a message listing TEST_USER, TEST_PASSWORD (names only). The masked line shows *** where the password was. The guard accepts the staging host and localhost and refuses the real shop. A guard like this is cheap insurance: one wrong BASE_URL in a CI variable should never let a "delete all items" test run against production. Note that the environment here is a fake object; real code would read os.environ in Python or System.getenv in Java.

Test accounts and test data

  • Create dedicated accounts for automation, clearly named ([email protected]), and never use a real person's account or real personal data.
  • Make tests clean up after themselves, or better, create the data they need through an API so each run starts from a known state.
  • Never run destructive or payment flows against production. Use a sandbox payment mode and test card numbers provided by the payment provider's documentation.
  • Do not bypass protections on your own site to make tests easier (for example, disabling a login lockout globally in production). Use a test-only configuration in a test environment instead.
  • CAPTCHAs are there to stop automation. In your own test environment, switch them off or use the vendor's documented test keys; do not build tools to defeat them.

Parallel runs make isolation even more important; see Running in CI for per-worker accounts.

Other people's sites: the rules

Automating a site you do not control is a different matter. Before you write a single locator, check three things.

Terms of service

Most sites state what automated access is allowed. Many forbid scraping, copying content at scale, or using bots to create accounts. A terms-of-service violation can lead to a blocked IP address or account, and in some cases to legal trouble. If the terms forbid what you want, do not do it, or ask for permission.

Is there an official way?

Before scraping, look for an API, a data export, a feed or a partnership program. An API gives you structured data, stable fields and a documented rate limit; a scraped page gives you fragile locators that break when the design changes. This is the strongest argument for not scraping at all.

robots.txt

robots.txt is a text file at the root of a site (https://example.com/robots.txt) in which the owner states which paths automated clients should not fetch. It is a convention rather than a technical barrier, and it is not a replacement for the terms of service, but respecting it is the baseline of polite automation. A file looks like this:

TEXT
User-agent: *
Disallow: /admin/
Disallow: /cart
Allow: /cart/help
Crawl-delay: 5

User-agent: BadBot
Disallow: /

The group for * applies to every client without a more specific group. Disallow blocks a path prefix, Allow makes an exception, and the longest matching rule wins. Crawl-delay is a hint some sites publish asking for a pause between requests (not every crawler honors it). The stand-in below is a simplified parser, enough to see the logic. Real parsers also handle wildcards (*, $) and many edge cases, so use a maintained library, such as Python's standard urllib.robotparser, for real work.

JS A simplified robots.txt checker

Press Run (or Ctrl + Enter) to see the console output here.

No output.

/cart is blocked but /cart/help is allowed, because the longer Allow rule beats the shorter Disallow. BadBot has its own group that blocks everything, while MyTestBot falls back to the * group. Remember that a missing robots.txt means "no stated restrictions", not "permission for anything"; the terms of service still apply.

Rate limits and being a good guest

A script is faster than any human, which is exactly the problem. Hundreds of requests per second can slow a small site down or get you blocked. Good behavior looks like this:

  • Throttle. Leave a pause (a second or more) between page loads, and follow any Crawl-delay or documented API limit.
  • Add a little randomness so your traffic is not a perfect metronome, which also looks less like an attack.
  • Back off on errors. If the server answers with HTTP 429 (Too Many Requests) or a 5xx error, wait longer each time instead of retrying immediately.
  • Identify yourself when you are allowed to, for example with a descriptive user agent and a contact address, so site owners can reach you.
  • Cache what you fetched and do not download the same page repeatedly.
JS A polite request schedule with backoff

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The schedule spaces five requests roughly two to two and a half seconds apart, using a seeded generator so the numbers are the same every run. The backoff doubles each time (1, 2, 4, 8, 16 seconds) and stops growing at a 20-second cap. At one request every two seconds a script makes 30 requests a minute, a rate a human could almost reach.

Why Selenium is not a scraping framework

Selenium was built to test web applications, and it shows when you use it to collect data at scale:

  • Heavy. Every session is a full browser with its own memory and CPU use. A plain HTTP client fetches a page in a fraction of the resources.
  • Slow. You wait for rendering, scripts and images that you do not need.
  • Fragile. Locators break whenever the page layout changes; there is no contract like an API has.
  • No crawling toolkit. Queues, de-duplication, retries, scheduling and storage are all things you would build yourself, while dedicated scraping frameworks provide them.

Use Selenium when the data truly appears only after complex JavaScript in a real browser, when you need to test your own application end to end, or for small, permitted tasks. For everything else, prefer an official API or a plain HTTP request. A small table-reading exercise appears in Project: Extracting Table Data.

Bot detection basics

Many sites try to tell humans from scripts. Common signals include request rates, missing or odd headers, browser properties that automation sets (for example, the navigator.webdriver flag in browsers controlled by WebDriver), repeated identical behavior, and challenges such as CAPTCHAs. When a site blocks your automation, take it as information: the owner does not want that traffic. Do not respond with tricks that hide the automation or impersonate people to get around a block on a site you do not own. If you test your own site, ask your team to allow-list your test traffic or provide a test environment without the challenge, as described above.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Where should a test's password normally come from?

    Secrets in source code end up in every clone and often in logs. Environment variables or a CI secret store keep them out of the repository.
  2. 2. In robots.txt, Disallow: /cart and Allow: /cart/help are both present. What happens to /cart/help/faq?

    The longest matching rule decides. /cart/help is longer than /cart, so the Allow rule applies to paths under it.
  3. 3. The site you want data from offers a documented API. What is the best choice?

    An official API is more reliable and more respectful than scraping HTML, and it avoids most legal and technical problems.
  4. 4. A server answers your script with HTTP 429. What should the script do?

    429 means you are sending too many requests. Backing off respects the limit; hiding or multiplying your traffic does not.

Summary

  • Keep secrets out of code and logs: environment variables or a secret store, with fail-fast checks and masking.
  • Use dedicated test accounts and test data, run only against allowed environments, and guard against a wrong BASE_URL.
  • For sites you do not own, read the terms of service, look for an official API first, and respect robots.txt.
  • Throttle requests, add some jitter, identify yourself where possible, and back off on 429 and 5xx responses.
  • Selenium is heavy and fragile for bulk data collection; it is a testing tool first.
  • A bot block is a signal to stop or ask permission, not a puzzle to defeat on someone else's site.

Next, put the testing skills to work in Project: Login Test Suite.