Skip to content
Tutorials Center
My learning

Topics

Lesson 13 of 36 ยท Traffic Management

Traffic Mirroring and Fault Injection

Test new versions with mirrored production traffic and test resilience with injected delays and aborts, using Istio VirtualService, plus runnable stand-ins.

Two features let you test with real traffic without hurting real users. Mirroring copies requests to a second destination and throws away its answers. Fault injection makes a call slow or failing on purpose to check that callers cope. Both are VirtualService fields, so you enable them with a config change and disable them the same way.

YAML here is read-only. The Python snippets are stand-ins illustrating the idea, not Istio.

Traffic mirroring (shadowing)

YAML
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: catalog
  namespace: shop
spec:
  hosts:
    - catalog
  http:
    - route:
        - destination: { host: catalog, subset: v1 }
      mirror:
        host: catalog
        subset: v2
      mirrorPercentage:
        value: 20.0

All users get answers from v1. For 20% of requests, the proxy also sends a copy to v2 "fire and forget": it does not wait for v2, and v2's response is discarded. Mirrored requests carry a modified Host/Authority header with -shadow appended so v2 can recognise them.

flowchart LR
    C["Client"] --> P["Proxy"]
    P -->|"real request"| V1["v1 (answers)"]
    P -.->|"copy, response dropped"| V2["v2 (shadow)"]
    V1 --> C

Users only ever see v1's answer; v2 sees real traffic shapes at no user risk.

What to remember:

  • Side effects are real. The shadow copy of POST /orders really runs in v2. Point the shadow at a test database or make it read-only, or you will create duplicate orders and send duplicate emails.
  • Mirrored traffic adds load: size v2 for the mirror percentage.
  • Compare v2 against v1 using metrics (error rate, latency) and logs; the user never sees differences, so you must look for them.

Fault injection

Faults are injected by the client-side proxy, so the callee code is untouched:

YAML
http:
  - match:
      - headers:
          x-chaos:
            exact: "on"
    fault:
      delay:
        percentage:
          value: 50.0
        fixedDelay: 3s
      abort:
        percentage:
          value: 10.0
        httpStatus: 503
    route:
      - destination: { host: inventory }
  - route:
      - destination: { host: inventory }
  • delay holds a request for fixedDelay before forwarding, simulating a slow dependency.
  • abort returns an HTTP status (or a gRPC status) without ever calling the destination, simulating a failure.
  • percentage says how many matching requests are affected.

Guarding the fault with a header match keeps experiments away from ordinary users: only requests carrying x-chaos: on are disturbed.

What to learn from an experiment: with a 3 s delay, does the web service time out at 2 s and show a friendly fallback, or does the whole page hang? With a 503 abort, are retries bounded? Do dashboards and alerts notice? Fault injection answers these questions before a real outage does.

Be careful when combining fault injection with retries and timeout on the same route: the interaction is subtle, and the documentation for your Istio version is the authority. Put the fault on the route the caller uses, and test the caller's own timeout and retry policy against it.

Stand-ins

Python Stand-in: a mirror that must not change the answer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Python Stand-in: does the caller survive a fault?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Safe habits

  1. Announce experiments; run them first in staging or on internal traffic.
  2. Use a header or sourceLabels match, and a small percentage.
  3. Have a one-command rollback: delete the fault block.
  4. Watch the golden signals while the fault is active.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What happens to the response of a mirrored request?

    Mirroring is fire-and-forget: the proxy does not wait for the shadow and drops its reply.
  2. 2. Why is mirroring a POST dangerous?

    The mirrored copy runs real code; use a test database or a read-only shadow.
  3. 3. What does an abort fault do?

    Abort fails the request immediately, simulating a failing dependency.
  4. 4. How can you keep a fault experiment away from ordinary users?

    A match limits the blast radius to requests you control.

Summary

  • Mirroring copies a share of traffic to another version and drops its responses.
  • Shadows have real side effects: isolate their data and size them for the load.
  • Fault injection adds delays or aborts at the proxy to test callers' resilience.
  • Guard faults with matches and small percentages, and have a quick rollback.
  • Compare metrics, because users never see mirror differences.

Next: Service Entries and Egress Control