Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 36 ยท Projects and Practice

Project: A Hardened Base Image Pipeline

Build a hardened base image pipeline: layered images, a hardening script verified by tests, vulnerability and secret scans as gates, weekly rebuilds, channels and an audit trail.

The previous project built one application image. Platform teams usually own something more fundamental: the hardened base image that every other image and server builds on. A flaw here affects everything, so the pipeline around it deserves the most care. In this project you design that pipeline end to end: build, harden, test, scan, promote, rebuild and retire.

Packer, scanners and cloud accounts cannot be used on this site. Configuration is read-only; the JavaScript stand-ins are policy gates you can adapt. All identifiers (CVE ids, account ids) are placeholders.

Requirements

  • A minimal OS base, patched on every build.
  • A documented hardening baseline, applied by a script or role and verified by automated tests.
  • Gates that stop a bad image: failed tests, high-severity vulnerabilities, secrets in the filesystem.
  • Weekly scheduled rebuilds plus on-demand rebuilds.
  • Channels with human approval for production, and a recorded audit trail.
  • Child images (runtime and application layers) automatically notified of a new base.

Stages

flowchart TD
  S["Schedule / merge / advisory"] --> V["fmt + validate + lint"]
  V --> B["Build base (patched)"]
  B --> H["Hardening applied"]
  H --> T["Spec tests (InSpec/Goss)"]
  T --> SC["Vulnerability + secret scan"]
  SC --> R["Register version + provenance"]
  R --> D["channel: dev"]
  D --> BT["Boot test"]
  BT --> ST["channel: staging"]
  ST --> A["Approval"]
  A --> P["channel: prod"]
  P --> C["Notify child image pipelines"]

Every box is a job or a gate; a failure at any stage stops promotion and leaves the previous prod version in place.

The hardening layer

The template installs updates, then applies a baseline. Keep the baseline in versioned scripts or an Ansible role, never hand-edited on a builder.

HCL
build {
  sources = ["source.amazon-ebs.base"]

  provisioner "shell" {
    inline = [
      "sudo apt-get update",
      "sudo DEBIAN_FRONTEND=noninteractive apt-get -y dist-upgrade"
    ]
  }

  provisioner "shell" {
    script          = "scripts/harden.sh"
    execute_command = "sudo -E bash '{{ .Path }}'"
  }

  provisioner "shell" {
    script          = "scripts/verify.sh"
    execute_command = "sudo -E bash '{{ .Path }}'"
  }

  provisioner "shell" {
    script          = "scripts/cleanup.sh"
    execute_command = "sudo -E bash '{{ .Path }}'"
  }

  post-processor "manifest" {
    custom_data = {
      version = var.version
      git_sha = var.git_sha
      parent  = data.amazon-ami.parent.id
    }
  }
}

data.amazon-ami.parent is a Packer data source that looks up the parent image, so you can record its id for ancestry. The verify.sh step runs the spec tests and fails on any violated control (testing images). What harden.sh configures is covered in hardening and securing images: SSH settings, sysctl, services removed, auditd, file permissions, banner, time sync.

Gates after the build

  1. Spec tests on the booted instance (InSpec profile for your baseline, or Goss).
  2. Vulnerability scan of the instance or image snapshot with your scanner; fail on high and critical findings that have a fix, and keep an expiring exception list for the rest.
  3. Secret scan of the filesystem for private keys and tokens.
  4. Provenance: record commit SHA, pipeline run, parent image id and scan report location in the registry or tags.

Report results as pipeline artifacts, so auditors can see why version 1.8.3 was promoted.

Controls for production

  • Approval: moving prod needs a second person or a protected environment.
  • Time windows and canaries: roll out to a canary group first (lifecycle lesson).
  • Revocation: a way to mark a version as bad and block new use.
  • Least privilege: the build role can create images and tags but not delete production resources; the promotion role can only change channels.

Try it: the promotion gate

JS Gate evaluation (stand-in)

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Try it: cascading child rebuilds

JS Which children need a rebuild? (stand-in)

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What should happen when the vulnerability scan finds a fixable critical issue?

    Gates stop risky images; exceptions must be explicit and time-limited.
  2. 2. Why record the parent image id in the manifest?

    Ancestry enables cascading rebuilds after a parent changes.
  3. 3. Who should be able to move the prod channel?

    Least privilege and approval keep production changes controlled.
  4. 4. Why run verification before the cleanup step?

    Tests need the build tools and state; cleanup then removes leftovers just before the snapshot. Boot tests later check the final image.

Summary

  • A base image pipeline is a sequence of gates: tests, scans, provenance, channels and approval.
  • Harden with versioned code, verify it with specs, clean up last.
  • Record the parent id so changes cascade to child images.
  • Keep rollback and revocation possible, and give each role only the permissions it needs.

Next: Practice: troubleshooting and reviewing Packer templates