Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 37 ยท Operating Microservices

Deployment Strategies and CI/CD

Release microservices safely: rolling, blue-green and canary deployments, per-service CI/CD pipelines, contract tests in CI, and a runnable canary simulator.

The promise of microservices is that each team can ship its own service on its own schedule. That promise only holds if shipping is safe, automatic and boring. If every release is a nervous midnight event, teams will release rarely, in big batches, and the independence you wanted is gone.

This lesson covers the two halves of safe delivery. CI/CD pipelines build, test and ship each service automatically. Deployment strategies (rolling, blue-green, canary) decide how a new version replaces the old one so that a bad release hurts few users, and can be undone fast.

Three ways to roll out

Rolling Replace instances one by one.
Blue-green Two full copies. Flip the switch.
Canary Send a small share of users first.

All three aim for zero downtime and a fast way back.

1 / 2

Per-service pipelines

In a monolith there is one pipeline for the whole product. In microservices, each service has its own pipeline, triggered when its own code changes. A change in payments builds and deploys only payments. This is the practical meaning of "independently deployable" (What Are Microservices?).

A typical pipeline has these stages:

  1. Build the code once and package it as an immutable image tagged with the commit (for example payments:9f3c1d2).
  2. Test: fast unit tests, then contract tests and component tests (Testing Microservices).
  3. Scan dependencies and the image for known vulnerabilities and leaked secrets.
  4. Deploy to staging and run smoke tests against the real running service.
  5. Deploy to production with a strategy from this lesson, then verify using metrics and health checks.

The same image moves through every stage; only the configuration changes (Configuration, Feature Flags and Secrets). A pipeline definition is code kept next to the service. Here is a real example in GitHub Actions syntax, read-only because the site cannot run CI:

YAML
name: payments
on:
  push:
    paths: ["services/payments/**"]
jobs:
  build-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: make -C services/payments test
      - run: make -C services/payments contract-test
      - run: docker build -t registry.example.com/payments:${{ github.sha }} services/payments
      - run: docker push registry.example.com/payments:${{ github.sha }}

The paths filter is the key microservices detail: it runs this pipeline only when files of this service change.

Contract tests in CI

The scary question in a pipeline is "will this change break the services that call mine?" Running every other service's tests is impractical. Instead, consumers publish contracts (what they send and what they expect back), and the provider's pipeline verifies that its new build still satisfies every published contract, before deploying. A failing contract check stops the release and names the consumer that would break. The same check can run in the consumer's pipeline against the provider's latest published version. The mechanics, and a tiny checker, are in Testing Microservices; evolving contracts safely is the subject of API Versioning and Evolving Contracts.

Rolling deployment

A rolling update replaces instances gradually. With four instances of version 1, the platform starts one of version 2, waits until it reports ready, stops one of version 1, and repeats. During the rollout both versions serve traffic, and the number of healthy instances never drops far, so users see no downtime.

Two settings control the pace, named as in Kubernetes: max surge (how many extra instances may exist during the update) and max unavailable (how many may be missing). The readiness probe is essential: without it the platform would send traffic to a new instance that is still starting up (Containers, Docker Compose and Kubernetes).

The simulation below runs a rolling update of four instances with a surge of one and at most one unavailable. A * marks a new instance that is not ready yet.

JS A rolling update, step by step

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The version mix changes one instance at a time, and the ready count never fell below four, because the old instance is only stopped after a new one is ready. The rollout is slow on purpose. The drawbacks are that two versions run together (your database and API changes must tolerate that, see API Versioning and Evolving Contracts) and that rolling back means rolling forward to the old version, which takes as long as the rollout.

Blue-green deployment

In blue-green, you keep two complete environments. Blue serves production. You deploy the new version to green, test it with no real users, and then switch the router so that all traffic goes to green. Blue stays up, idle, for a while.

  • Pros: the switch is instant, and rollback is just switching back. You can test the real environment before users see it.
  • Cons: you pay for double the capacity during the release, and databases are awkward: both colors usually share one database, so schema changes must work with both versions (the expand-and-contract technique).

Blue-green gives you an all-or-nothing flip: either nobody or everybody sees the new version. When you want to learn from real traffic before committing, use a canary.

Canary deployment

A canary release sends a small share of real traffic, say 5%, to the new version while the rest stays on the stable one. You compare their metrics: error rate, latency, business numbers. If the canary looks healthy, you raise the share (25%, 50%, 100%). If it looks worse, you route everyone back and only 5% of users were affected.

This depends on the observability from Observability: Structured Logs and Metrics: you cannot judge a canary without metrics split by version. The traffic split happens at a load balancer, service mesh or gateway (API Gateway and Backend for Frontend), often with sticky routing so a user does not bounce between versions. Many teams automate the whole loop: step up the weight, check the numbers, and roll back automatically.

The stand-in below is a canary traffic splitter with automatic rollback. It routes simulated requests by weight, tracks error rates per version, and applies a simple rule: if the canary's error rate is more than two percentage points above stable (with at least 50 requests seen), roll back. A seeded random generator makes the output repeatable.

JS Canary releases with automatic rollback

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The good release passes every step; its error rate stays near the stable version's, with small differences from random noise. The bad release is caught at the first step: with only about 4% of traffic (174 of 4,000 requests) it showed roughly a 10% error rate against about 1% for stable, so the rule rolled it back and about 96% of users never saw it. Notice the minimum sample size: with too few requests, the rate is noise, and a real system needs a larger sample or a statistical test before deciding.

Choosing a strategy

Strategy Downtime Rollback speed Extra cost Best for
Rolling None Slow (roll forward again) Low Everyday releases, backward-compatible changes
Blue-green None Instant Double capacity briefly Risky releases where you want a quick undo
Canary None Fast Needs traffic split and metrics High-traffic services, risky behavior changes

They combine well with feature flags (Configuration, Feature Flags and Secrets): deploy the code to everyone with the flag off, then release the feature gradually. Deploying and releasing become separate decisions, which is the safest model.

Database changes and rollbacks

The database is where deployments get stuck, because you cannot roll a schema back as easily as an image. Use expand and contract: first expand (add the new column or table, keep the old), deploy code that writes both and reads the new, migrate the data, and only later contract (remove the old). Each step is compatible with the previous version of the code, so any step can be rolled back. Never deploy a migration that breaks the version currently running.

Also decide your deployment frequency target and measure it. The DORA research metrics (deployment frequency, lead time for changes, change failure rate and time to restore service) are a common way to tell whether your delivery pipeline is healthy.

What is real and what is the stand-in

Rolling updates, blue-green, canaries and per-service pipelines are standard practice, and the YAML above is real syntax. The site cannot run CI, containers or a load balancer, so the TryIts simulate the decisions: the replacement order of a rolling update and the weight-and-rollback rule of a canary. Real platforms (Kubernetes, Argo Rollouts, service meshes, cloud load balancers) implement these with health probes, many more signals and gradual traffic shifting.

Make the fast path the safe path. If rolling back takes one button or happens automatically, teams ship more often, with smaller changes, and each one is easier to reason about.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does it mean that each microservice has its own pipeline?

    Independent pipelines are what makes independent deployment real. A path filter ensures only the changed service is built.
  2. 2. Why does a rolling update need readiness probes?

    The platform waits for the new instance to report ready before stopping an old one, which keeps capacity up.
  3. 3. What is the main advantage of blue-green over a rolling update?

    Both environments are complete, so you flip traffic at once and flip back if needed. The price is double capacity for a while.
  4. 4. In a canary release, what decides whether to promote the new version?

    A canary only works with observability: you compare real-traffic metrics by version and promote or roll back.
  5. 5. What does the expand-and-contract technique help with?

    You add the new structure first, migrate, and remove the old only once nothing uses it, so rollbacks stay possible.

Summary

  • Each service gets its own pipeline: build once, test (including contract tests), scan, deploy to staging, deploy to production, verify.
  • Rolling updates replace instances gradually, guarded by readiness probes; two versions run side by side.
  • Blue-green keeps two environments and flips traffic, giving instant rollback at double cost.
  • Canaries send a small share of real traffic to the new version and promote or roll back from metrics.
  • Combine strategies with feature flags and expand-and-contract database changes so every step can be undone.

Next: Serverless Functions and Microservices.