Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 38 ยท Projects and Practice

Project: A Build-Pipeline DSL in Groovy

Design and build a small pipeline DSL in Groovy with closure delegates: stages, steps, dependencies, validation, a script runner using GroovyShell, and Spock tests.

You have used DSLs such as Gradle and Spock. In this project you build one from scratch: a tiny pipeline language for describing build and deploy jobs. Writing it ties together closure delegates, @DelegatesTo, methodMissing, validation, GroovyShell and tests. The result reads like a configuration file but is real Groovy, so loops and variables work inside it.

Groovy cannot run on this site, so the Groovy sources are read-only. A Java port at the end runs the same model: stages in order, dependency checks and a dry-run printout.

Design first: write the usage

Write what you want users to type before writing any implementation:

GROOVY
pipeline "shop-service" {
  env JAVA_HOME: "/opt/jdk", STAGE: "prod"

  stage "build" {
    step "compile", "gradle compileJava"
    step "unit tests", "gradle test"
  }

  stage "package", after: "build", {
    step "jar", "gradle jar"
  }

  stage "deploy", after: ["build", "package"], {
    ["eu", "us"].each { region ->
      step "deploy ${region}", "deploy.sh ${region}"
    }
  }
}

Needs: a top-level pipeline entry point, env taking a map, stage with an optional after and a body, step with a name and a command. Notice the loop in the last stage. That is free because the body is a closure.

The model

GROOVY
class Step {
  String name
  String command
}

class Stage {
  String name
  List<String> after = []
  List<Step> steps = []
}

class Pipeline {
  String name
  Map<String, String> env = [:]
  List<Stage> stages = []
}

Plain classes. They know nothing about the DSL.

The builders (delegates)

GROOVY
class StageBuilder {
  final Stage stage

  StageBuilder(Stage stage) { this.stage = stage }

  void step(String name, String command) {
    stage.steps << new Step(name: name, command: command)
  }
}

class PipelineBuilder {
  final Pipeline pipeline

  PipelineBuilder(String name) { pipeline = new Pipeline(name: name) }

  void env(Map<String, String> vars) { pipeline.env.putAll(vars) }

  void stage(Map opts = [:], String name,
             @DelegatesTo(value = StageBuilder, strategy = Closure.DELEGATE_FIRST) Closure body) {
    def stage = new Stage(name: name)
    stage.after = [opts.after ?: []].flatten()
    def builder = new StageBuilder(stage)
    def code = body.rehydrate(builder, this, this)
    code.resolveStrategy = Closure.DELEGATE_FIRST
    code()
    pipeline.stages << stage
  }
}

Pipeline pipeline(String name,
                  @DelegatesTo(value = PipelineBuilder, strategy = Closure.DELEGATE_FIRST) Closure body) {
  def builder = new PipelineBuilder(name)
  def code = body.rehydrate(builder, null, null)
  code.resolveStrategy = Closure.DELEGATE_FIRST
  code()
  Validator.check(builder.pipeline)
  builder.pipeline
}

Notes on the design:

  • A named-argument call such as stage "package", after: "build", { ... } passes the map first, which is why stage takes Map opts = [:] before the name.
  • rehydrate creates a copy of the closure with the right delegate, so a closure can be reused and the builders are safe across threads.
  • @DelegatesTo gives IDE completion, and @CompileStatic can check the DSL scripts.
  • The strategy is DELEGATE_FIRST, so step resolves in the StageBuilder.

Validation

Errors should be about the DSL's concepts, not about Java stack traces.

GROOVY
class Validator {
  static void check(Pipeline p) {
    def names = p.stages*.name
    def dupes = names.findAll { n -> names.count(n) > 1 }.unique()
    if (dupes) throw new IllegalStateException("Duplicate stage names: ${dupes.join(', ')}")

    p.stages.each { st ->
      st.after.each { dep ->
        if (!(dep in names)) {
          throw new IllegalStateException("Stage '${st.name}' runs after unknown stage '${dep}'")
        }
      }
      if (!st.steps) throw new IllegalStateException("Stage '${st.name}' has no steps")
    }
  }
}

Ordering the stages

A pipeline must run each stage after those it depends on. This is a topological sort:

GROOVY
List<Stage> executionOrder(Pipeline p) {
  def done = [] as LinkedHashSet<String>
  def byName = p.stages.collectEntries { [it.name, it] }
  def visiting = [] as Set

  def visit
  visit = { String n ->
    if (n in done) return
    if (!visiting.add(n)) throw new IllegalStateException("Cycle at stage '${n}'")
    byName[n].after.each { visit(it) }
    visiting.remove(n)
    done << n
  }
  p.stages*.name.each { visit(it) }
  done.collect { byName[it] }
}

The closure visit calls itself, so it is declared first (def visit) and assigned afterwards.

flowchart LR
  A["build"] --> B["package"]
  A --> C["deploy"]
  B --> C

A valid order is build, package, deploy. A cycle would be reported as an error.

Running scripts from a file

The DSL files are scripts. Users write shop.pipeline, and your tool loads it:

GROOVY
import org.codehaus.groovy.control.CompilerConfiguration

def config = new CompilerConfiguration()
config.scriptBaseClass = PipelineScript.name          // a Script subclass that holds pipeline()

def shell = new GroovyShell(this.class.classLoader, new Binding(), config)
def pipeline = shell.evaluate(new File("shop.pipeline"))

executionOrder(pipeline).each { st ->
  println "== ${st.name}"
  st.steps.each { println "   ${it.name}: ${it.command}" }
}

A script base class lets the file call pipeline "x" { ... } directly, because the method is found on the script itself. For untrusted files, restrict the language with a SecureASTCustomizer, as noted in the lesson on Groovy with Java and Spring.

Testing the DSL with Spock

GROOVY
class PipelineDslSpec extends Specification {

  def "builds stages in order"() {
    when:
    def p = pipeline("demo") {
      stage("a") { step "x", "echo x" }
      stage("b", after: "a") { step "y", "echo y" }
    }

    then:
    executionOrder(p)*.name == ["a", "b"]
  }

  def "rejects unknown dependencies"() {
    when:
    pipeline("demo") {
      stage("b", after: "ghost") { step "y", "echo y" }
    }

    then:
    def e = thrown(IllegalStateException)
    e.message.contains("unknown stage 'ghost'")
  }
}

Extensions to try

  1. A parallel { } block inside a stage.
  2. when: { env.STAGE == "prod" } conditions on stages.
  3. A --dry-run flag that prints commands instead of running them with ["sh", "-c", cmd].execute().
  4. Variables shared across steps through a Binding.
  5. Load several pipeline files and report which ones are invalid.

Try it: the model, validation and ordering in Java (stand-in)

This program builds a pipeline with nested lambdas (the closures), validates it, and prints the topological order.

Java Stand-in: pipeline model with validation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Try it: detecting a cycle (stand-in)

Dependencies that loop can never be ordered. This small program finds the cycle the way visiting does.

Java Stand-in: cycle detection

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why design the DSL usage before writing the implementation?

    A DSL is for its readers. Starting from the text they will write keeps the design focused on them.
  2. 2. What does body.rehydrate(builder, owner, thisObject) give you?

    rehydrate returns a clone with a new delegate, owner and this, so you do not mutate a shared closure.
  3. 3. How is the dependency order of stages computed?

    Each stage is visited after the stages it depends on, and a revisit during traversal signals a cycle.
  4. 4. Where should DSL errors point?

    Readable validation messages are part of a good DSL.

Summary

  • Write the intended usage first, then model, builders, validation and execution.
  • Builders are delegates; rehydrate, DELEGATE_FIRST and @DelegatesTo make them robust and tool-friendly.
  • Validate when the block ends and report problems in the DSL's own words.
  • Sort stages topologically and detect cycles.
  • Load DSL files with GroovyShell and a script base class, and test with Spock.

Next: Groovy Practice Exercises