Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 36 ยท Projects and Practice

Project: Spike and Soak Tests with SLOs

Turn service level objectives into k6 thresholds, design a spike test and a soak test, and use runnable Node simulations to detect slow recovery after a spike and a memory-leak trend.

The first project proved that a system meets its goals under normal peak load. Two harder questions remain: what happens when traffic suddenly triples (spike), and does the system stay healthy for hours (soak). In this project you translate SLOs into thresholds, write both tests in k6 and use simulations to practice reading the results.

Honesty note. The k6 scripts are real but read-only (k6 cannot run on this site). The runnable blocks are deterministic Node simulations of a queue and a leaking service, built so you can practice the analysis. They are not measurements of a real system or of k6.

From SLOs to thresholds

An SLI (service level indicator) is something you measure, such as the share of requests faster than 300 ms. An SLO (objective) is the target for it, such as "99.5 % of requests succeed and 95 % are faster than 300 ms over 30 days". The gap between 100 % and the objective is the error budget.

Load tests translate this directly:

SLO k6 threshold
99.5 % success http_req_failed: ['rate<0.005']
95 % of requests under 300 ms http_req_duration: ['p(95)<300']
Checkout under 800 ms for 99 % 'http_req_duration{name:Checkout}': ['p(99)<800']
Every page returns the right content checks: ['rate>0.995']

Use stricter thresholds for a load test than for the monthly SLO if you want margin, and looser ones for spike tests, where degradation is acceptable as long as the system recovers.

Node.js Stand-in: how much error budget did a test burn?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Part 1: the spike test

A spike jumps to a high load almost instantly, holds briefly and drops. The goals differ from a load test: you want to see how bad it gets and how fast the system recovers, not whether it stays under p(95) 300 ms the whole time.

JavaScript
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  scenarios: {
    spike: {
      executor: 'ramping-arrival-rate',
      startRate: 20, timeUnit: '1s',
      preAllocatedVUs: 200, maxVUs: 1500,
      stages: [
        { target: 20, duration: '2m' },    // normal
        { target: 200, duration: '20s' },  // sudden 10x jump
        { target: 200, duration: '1m' },   // hold the spike
        { target: 20, duration: '20s' },   // drop
        { target: 20, duration: '5m' },    // RECOVERY window: the part that matters
      ],
    },
  },
  thresholds: {
    // Looser during the spike is fine; failures should stay modest:
    http_req_failed: ['rate<0.05'],
    // The recovery window is tagged by the stage via a custom tag in the script, see below.
    'http_req_duration{phase:recovery}': ['p(95)<400'],
  },
};

export default function () {
  // Tag requests with the phase, derived from the elapsed time of the test.
  const res = http.get(`${__ENV.BASE_URL}/api/products`, { tags: { phase: phaseNow() } });
  check(res, { 'status 200': (r) => r.status === 200 });
}

Where phaseNow() compares Date.now() - testStart against your stage boundaries (compute testStart in setup() and return it). The key idea is to tag the recovery window so a threshold can judge it on its own. After the spike ends, a healthy system returns to its baseline within a minute or two. Signs of trouble: latency staying high after the load drops (queues still draining, connection pools exhausted, autoscaling slow, caches evicted), or errors continuing for minutes.

Also watch the arrival-rate scenario's dropped_iterations (Built-in metrics): if it appears during the spike, your generator or maxVUs could not deliver the intended rate.

Node.js Simulation: queue backlog after a spike and time to recover

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The simulation shows a general truth: a backlog built during the spike takes longer to drain than the spike lasted, because the spare capacity (here 50 req/s) is only half of the server's capacity. A 20-second spike can hurt for over a minute.

Part 2: the soak test

A soak test holds realistic load for hours. It looks for problems that need time: memory leaks, file-descriptor or connection leaks, log disks filling, caches growing without bound, slow performance decay, expiring tokens and certificates, scheduled jobs (backups, batch runs) colliding with traffic.

JavaScript
export const options = {
  scenarios: {
    soak: {
      executor: 'constant-arrival-rate',
      rate: 30, timeUnit: '1s',
      duration: '4h',
      preAllocatedVUs: 100, maxVUs: 400,
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.005'],
    http_req_duration: ['p(95)<300'],
  },
  // Keep results small over a long run: drop per-URL tags and bodies.
  discardResponseBodies: true,
  systemTags: ['status', 'method', 'scenario', 'name'],
};

Considerations specific to long tests:

  • Tokens expire. Refresh them (Authentication and Correlation).
  • Output size. A four-hour run produces huge point data; use an aggregating backend instead of --out json.
  • A flat average hides a trend. The test can pass its thresholds yet show latency creeping upward each hour. Always graph over time.
  • Watch the server, not only k6: memory, GC time, open connections and disk usage over the whole period.
Node.js Simulation: detect a latency creep with a trend line

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Both services pass the 300 ms threshold for the whole run, yet one is clearly leaking. This is why the soak report needs a trend (slope) check or a person looking at the graph: pass/fail alone is not enough. Extrapolate: at 15 ms per hour of creep, the leaking service breaches 300 ms after 6 hours of production uptime.

Packaging the tests

Store both scripts in the repository next to the smoke and load tests. Run the spike test weekly or before expected events (a launch, a sale), the soak test nightly or weekly in a dedicated environment, and review the trend dashboards. See k6 in CI pipelines for scheduling and Analyzing Results for the interpretation method.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main question a spike test answers?

    Spike tests focus on behaviour during the surge and on recovery afterwards.
  2. 2. If an SLO allows 0.5% failures, which threshold expresses it?

    http_req_failed is a Rate; 0.5 percent is 0.005.
  3. 3. Why can a soak test pass all thresholds and still reveal a problem?

    Pass/fail looks at limits, not direction. Graph over time and measure the slope.
  4. 4. Why tag requests with a phase (such as recovery) in a spike test?

    A tag creates a sub-metric, so you can demand good latency after the spike without demanding it during the surge.

Summary

  • SLOs map onto thresholds; the remainder is the error budget.
  • Spike tests judge behaviour during the surge and the recovery time afterwards.
  • Soak tests need trends over time, token refresh and compact outputs.
  • The Node simulations let you practice recognizing backlog drain and latency creep.

Next: Project: A Performance Gate in CI