Skip to content
Tutorials Center
My learning

Topics

Lesson 46 of 60 ยท Automation Concepts and Process

When and What to Automate

Decide what to automate and when: ROI and break-even, good candidates, the test pyramid, flaky tests, maintainability and tools, with runnable calculators.

Sooner or later someone says, "Let's automate all our testing." It sounds efficient, and sometimes it is. But automation is software: it takes time to write, breaks when the product changes, and needs care for as long as it lives. Teams that automate the wrong things end up with a slow, fragile suite that nobody trusts and everybody ignores. Teams that automate the right things get fast feedback on every change and free their testers to do the creative work that only humans can do.

This lesson is about the decision, not the tools: what to automate, what to leave manual, how to estimate whether automation will pay back, and how to keep a suite healthy. The calculators are stand-ins for the spreadsheet a team would build; the real automation frameworks are taught in their own tracks, linked at the end.

What automation is good at, and what it is not

A script is excellent at doing the same thing precisely, quickly and tirelessly. A human is excellent at noticing, judging and improvising. Choose accordingly.

Machines are good at

Repetition The same check on every build.
Speed Thousands of checks in minutes.
Precision Exact values, no tired eyes.
Any time Nights, weekends, every commit.
1 / 2

A useful way to say it: automation does checking (comparing results with expectations you already wrote down), while people do testing (learning about the product and finding what you did not think to check). Both are needed; see exploratory testing in practice.

Good candidates and poor candidates

Good candidates are tests that are repeated often, stable, deterministic and costly to run by hand:

  • Regression checks for important behavior that must never break.
  • Smoke tests that run on every build to confirm the basics work.
  • Data-driven tests that run the same steps with many inputs, such as boundary values from boundary value analysis.
  • Calculations and business rules, like pricing and tax, which are exact and easy to check by machine.
  • Cross-configuration runs, such as the same suite on several browsers or operating systems.
  • Load and API checks that are impractical for a person to perform.

Poor candidates are the opposite:

  • Features that are still changing every week. The script breaks as fast as you write it.
  • One-off or rarely run tests. Writing costs more than running by hand.
  • Tests about look and feel, such as "does this layout seem balanced?"
  • Exploratory and usability sessions, which depend on human judgment.
  • Tests that need physical interaction (a card reader, a gesture on real hardware) unless the lab is built for it.
  • Tests with unreliable or unpredictable results, which will be flaky until the cause is fixed.

A common failure is to automate every manual test case as is. Many manual cases are long, vague stories ("check the whole checkout flow") that make poor scripts. Good automated tests are small, focused and independent, with one reason to fail.

The cost-benefit question: a break-even calculation

Automation has an upfront cost (building the script and framework) and a running cost (maintenance and the time to investigate failures), and it saves the cost of manual execution every time the test would have been run. The break-even point is the number of runs after which automation becomes cheaper.

A simple model, for one test (or a group of tests):

  • build = hours to write the automated test
  • manual = hours to run it by hand once
  • auto = hours of human attention per automated run (usually near zero)
  • maintain = hours per month to keep the script working
  • runsPerMonth = how often it would be run

Savings per month = runsPerMonth * (manual - auto) - maintain. Break-even months = build / savings when savings are positive.

JS Break-even calculator

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The smoke test pays for itself in a matter of days (0.1 months). The pricing rules take about 0.4 months, still well under a month. The quarterly report test and the volatile UI test never pay off with these numbers: the first because it runs too rarely, the second because maintenance eats the savings. The numbers are invented, but the pattern is real: frequency and stability drive the return.

Be honest about what the model leaves out. Automation also gives value that is not hours saved: faster feedback to developers, the confidence to change code, and checks that run at night. On the other side, the initial framework, tools, training, infrastructure and time spent investigating failures often cost more than the first estimate. Treat results as a guide for discussion, not a guarantee.

The test pyramid

Where to automate matters as much as what. The test pyramid is a rule of thumb: have many fast, small tests at the bottom, fewer in the middle, and a handful of slow, broad tests at the top.

flowchart TB
  A["UI / end-to-end tests<br/>few, slow, broad"] --> B["API / integration tests<br/>some, medium speed"]
  B --> C["Unit tests<br/>many, fast, focused"]

Wide base, narrow top: put most automation where it is cheap and reliable.

  • Unit tests check one function or class in isolation: milliseconds each, rarely flaky, and pinpoint the faulty code. Developers usually write them (see test levels).
  • API and integration tests check that components work together through their interfaces. They are slower than unit tests but much more stable than screen-based tests.
  • UI end-to-end tests drive the real interface. They give the most realistic answer but are the slowest, most fragile and hardest to diagnose.

A suite that is mostly end-to-end tests (the "ice cream cone") is slow, expensive and flaky. If a rule can be checked in a unit or API test, check it there, and keep UI tests for the critical journeys that really need a browser. Some teams prefer a "trophy" or "honeycomb" shape that emphasizes integration tests; the shared lesson is to favor fast and focused checks.

Flaky tests

A flaky test sometimes passes and sometimes fails without any change to the code. Causes include timing assumptions, shared test data, order dependencies, unstable environments and real network calls. Flakiness is poisonous: people stop trusting red builds, re-run until green, and real bugs slip through.

The arithmetic is brutal. If each test is independently flaky with a small probability, the chance that a whole run is clean drops fast as the suite grows.

JS How flaky tests add up

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Even a seemingly tiny flake rate of half a percent per test leaves a 500-test suite green only about 8 percent of the time. The lesson: treat flakiness as a defect in the test suite. Quarantine or fix flaky tests quickly, never let them pile up, and design tests to be independent, with their own data and no fixed sleeps. For tool-specific advice see flaky tests in Cypress and Playwright.

Keeping automation maintainable

  • Keep tests small and independent. One behavior per test, no hidden dependence on another test's leftovers.
  • Name tests by behavior. A failure name like rejects expired discount code tells the story without opening the code.
  • Separate what from how. Wrap fragile details (selectors, URLs) in one place, such as a page object, so a UI change means one edit.
  • Use reliable test data. See test data and environments.
  • Review test code like product code.
  • Delete tests that no longer add value; a suite is not a trophy collection.
  • Watch the trend. Track run time, failure rate and flaky count, and act when they drift.

The tool landscape (and where to learn it)

Choosing tools comes after choosing what to automate. The common families are:

When picking, favor what your team already knows, what fits your tech stack, and what integrates with your build pipeline. A modest tool used well beats a fashionable one used badly.

The automation test life cycle

Automation is a mini project inside the test project. It has a life cycle of its own, and teams that skip steps end up with a suite nobody trusts. The usual stages:

  1. Decide the scope. Which tests are worth automating, and which are not (use the criteria earlier in this lesson: repeatable, stable, high-value, data-heavy, run often). Write down the goals: faster releases, fewer regression escapes, nightly coverage.
  2. Choose the approach and tools. Pick the test level (unit, API, UI), the language and framework, the data approach and the reporting, based on the team's skills and the technology (Automation Frameworks, Selenium, Cypress and Playwright). Run a short proof of concept against real screens before committing.
  3. Plan and design. Define the framework structure, naming, test data strategy, environments and how results are reported. Design tests to be independent and repeatable.
  4. Build the framework and the tests. Start with a thin vertical slice (one test that runs end to end in the pipeline), then grow. Review automation code like production code.
  5. Execute. Run locally, then on a schedule or on every change in CI. Analyse failures: real defect, test defect, environment or data problem? Record the cause.
  6. Maintain. Update tests when the product changes, fix flaky tests, remove obsolete ones, upgrade tools and browsers. Maintenance is usually the largest cost, so budget for it.
  7. Report and improve. Track trends: pass rate, flakiness, run time, defects found by automation, effort saved. Use them to decide what to automate next.
flowchart LR
  S["Scope and goals"] --> T["Tools and proof of concept"] --> D["Design"] --> B["Build"] --> E["Execute in CI"] --> M["Maintain"] --> R["Report and improve"]
  R --> S

It is a loop: reports feed the next round of scoping.

Maintenance traps

  • Flaky tests that pass and fail without a code change teach the team to ignore red builds. Quarantine them, fix the cause (timing, shared data, order dependence), then restore them.
  • Duplicated steps copied across scripts make every UI change expensive.
  • Test data rot: accounts expire, records change. Create fresh data per run.
  • Too many UI tests: keep to the pyramid. Slow suites get skipped.
  • No owner: a suite without an owner decays within months.

Continuous integration and CI tools

Automated tests earn their value when they run without anyone remembering to run them. Continuous integration (CI) means merging code often into a shared branch and having a server build and test every change automatically. Continuous delivery extends that to a pipeline that can release at any time.

A typical pipeline for a web app:

  1. Commit or pull request triggers the pipeline.
  2. Build the application.
  3. Fast tests: linters and unit tests (seconds to minutes).
  4. Integration and API tests against a test database or containers.
  5. Deploy to a test environment automatically.
  6. Smoke tests, then selected UI and end-to-end tests, perhaps in parallel.
  7. Reports and gates: publish results; a failing gate blocks the merge or release.
  8. Nightly jobs: the full regression suite, performance checks, security scans.

The CI server provides the trigger, a clean environment, the schedule, parallelism, test reports, notifications and an audit trail.

Tool Notes
Jenkins Open-source automation server; huge plugin ecosystem; you host and maintain it; pipelines defined in a Jenkinsfile
GitLab CI/CD Built into GitLab; pipeline defined in .gitlab-ci.yml; runners execute jobs
GitHub Actions Built into GitHub; workflows in YAML under .github/workflows
Azure DevOps Pipelines, CircleCI, TeamCity, Bamboo, Travis CI Other hosted or self-hosted options

All of them share the same ideas: stages and jobs, triggers, artifacts (reports, screenshots, logs), environment variables and secrets, and parallel jobs. Learning one makes the others easy.

Practical tips for testers in CI:

  • Keep the fast feedback path (unit tests) under about ten minutes; push slow suites to nightly runs or run them in parallel.
  • Publish JUnit-style XML or HTML reports and keep screenshots and videos from failed UI tests.
  • Never put real credentials in the repository; use the CI secret store.
  • Make failures actionable: a clear name, the failing step and the evidence.
  • Track flaky tests and fix them within days.
JS A fail-fast pipeline simulation

Press Run (or Ctrl + Enter) to see the console output here.

No output.

This is the shift-left idea from TDD vs BDD and Shift-Left in practice: cheap checks first, expensive ones later, and a failure stops the line. The Agile Testing and CI lesson shows how the whole team uses the pipeline.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which test is the BEST candidate for automation?

    Frequent, stable and exact checks pay back quickly. The other options are one-off, judgment-based or exploratory.
  2. 2. In the test pyramid, which tests should be the most numerous?

    Unit tests are fast, cheap and precise, so most automated checks should live there, with fewer API and UI tests above them.
  3. 3. A test sometimes fails for no clear reason and passes on re-run. What is it called, and what should the team do?

    Flakiness erodes trust in the suite. Re-running until green hides the problem, so find the cause or quarantine the test.
  4. 4. A test costs 10 hours to automate and saves 0.5 hours per manual run, but maintenance costs 3 hours a month and it runs 4 times a month. What happens?

    Savings are 4 x 0.5 = 2 hours a month, but maintenance costs 3 hours, so the result is negative every month.

Summary

  • Automation is good at fast, exact, repeated checks; people are better at exploring, judging and learning.
  • Good candidates are stable, frequent and deterministic (regression, smoke, data-driven, calculations); poor ones are volatile, rare or judgment-based.
  • Estimate return with build cost, manual cost, maintenance and frequency; frequency and stability drive the break-even point.
  • Follow the test pyramid: many unit tests, fewer API tests, a few UI journeys.
  • Flaky tests are defects in the suite; fix or quarantine them quickly, and keep tests small, independent and reviewed.

Next: Testing in Agile Teams and CI/CD.