Skip to content
Tutorials Center
My learning

Topics

Lesson 39 of 65 ยท Reliability and Operations

Deployments and CI/CD

Learn CI/CD pipelines and safe release strategies for system design: rolling, blue-green and canary deploys, feature flags, rollbacks and zero-downtime database migrations.

Many outages are not caused by hardware or traffic spikes. They are caused by us: a change was shipped that broke something. Because large systems change many times a day, how you release is as important as what you build. CI/CD automates building, testing and delivering changes, and deployment strategies limit how much harm any one change can do.

What CI/CD means

Continuous Integration (CI): every change is merged into a shared branch frequently and automatically built and tested. Small, frequent merges mean conflicts and bugs surface within minutes instead of weeks.

Continuous Delivery: every change that passes the pipeline could be released at any time; a human presses the button.

Continuous Deployment: every change that passes is released to production automatically.

A typical pipeline:

flowchart LR
  C[Commit] --> B[Build]
  B --> U[Unit tests]
  U --> S[Static analysis and<br/>security scan]
  S --> I[Integration tests]
  I --> A[Artifact: image]
  A --> ST[Deploy to staging]
  ST --> P[Deploy to production<br/>gradually]
  P --> M[Monitor and auto-rollback]

Fast checks run first; the same artifact moves through environments; production rollout is gradual and watched.

Good practice: build once, promote the same artifact (for example a container image) through environments instead of rebuilding; keep configuration outside the artifact; make the pipeline fast (minutes, not hours) and keep the main branch always releasable.

Deployment strategies

Four ways to roll out

Rolling Replace instances a few at a time.
Blue-green Two full environments, flip the switch.
Canary Send a small share of traffic to the new version.
Feature flags Ship code dark, turn on later.

Each limits the blast radius of a bad release.

1 / 2

Recreate is the naive method: stop the old version, start the new one. It causes downtime and is only acceptable for non-critical systems.

Rolling update: instances are replaced a few at a time while the rest keep serving. No extra environment is needed. During the rollout both versions run together, so they must be compatible (same API and database schema). Rollback is another rolling update.

Blue-green: run two identical environments. Blue serves production; deploy the new version to green, test it, then switch the load balancer to green in one step. Rollback is switching back, which is instant, but you pay for double capacity and need to handle the database carefully, since both share it.

Canary: release to a tiny fraction (1-5%) of traffic or a single region, compare metrics against the old version, then widen gradually: 5%, 25%, 50%, 100%. If errors or latency rise, stop and roll back. It catches problems that tests miss, using real traffic, with limited damage.

Python A canary rollout with automatic rollback

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The bad release is caught when only a small share of users saw it. Automate this decision: tools such as Argo Rollouts, Flagger and cloud deploy services promote or roll back based on metrics.

Feature flags

A feature flag is a runtime switch that controls whether code runs. You can merge and deploy unfinished work turned off, enable it for employees, then for 1% of users, then for everyone, and turn it off instantly without a deploy. This separates deploy (code on servers) from release (users see it).

Python Percentage rollout with a feature flag

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Hashing the user ID gives each user a stable bucket, so a person does not flicker between versions. Remove old flags once a rollout is finished, or they become hidden complexity.

Database changes without downtime

Code rolls back easily; data does not. Since old and new code overlap during a rollout, schema changes must be compatible with both. The safe approach is expand and contract (parallel change):

  1. Expand: add the new column or table (nullable, with a default). Old code ignores it.
  2. Migrate: deploy code that writes to both old and new, and backfill existing rows in batches.
  3. Switch: deploy code that reads from the new place.
  4. Contract: after the old code is gone and you are sure, drop the old column.

Never rename or drop a column in one step while old code is still running. Keep migrations small and reversible where possible, and run large backfills in the background with throttling.

Rollback and recovery

  • Keep the previous version ready and make rollback a single command that you have practiced.
  • Prefer roll forward with a quick fix when rollback is unsafe (after an irreversible migration).
  • Use immutable infrastructure: replace servers with new images instead of patching them in place; it removes configuration drift.
  • Use infrastructure as code (Terraform, Pulumi, Kubernetes manifests) so environments are reproducible and reviewed like code.
  • Deploy small, often. A change of ten lines is easier to review, test and revert than one of ten thousand.
  • Avoid risky times (Friday evening) and have monitoring from the previous lesson watching every release.

Platforms such as Kubernetes provide rolling updates and health probes out of the box; see Docker for containers, the usual unit of deployment.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main advantage of a canary release?

    A canary limits the blast radius and uses real traffic to reveal problems that tests missed.
  2. 2. How does rollback work in a blue-green deployment?

    The old environment is still running, so flipping traffic back is nearly instant.
  3. 3. Why separate deploy from release with feature flags?

    Flags control exposure at runtime, so you can release gradually and switch off without a new deploy.
  4. 4. What is the first step of expand-and-contract for renaming a column?

    Old and new code run side by side during rollouts, so first add the new column compatibly and migrate step by step.

Summary

  • CI builds and tests every change; delivery and deployment automate releasing it.
  • Build once and promote the same artifact; keep pipelines fast and main always releasable.
  • Rolling, blue-green and canary strategies trade cost, speed and risk; canaries catch problems on real traffic.
  • Feature flags decouple deploy from release and provide kill switches.
  • Change databases with expand-and-contract and keep rollbacks practiced.

Next: Security for System Design