Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 40 ยท Projects

Project: Build a Mini Jenkins Pipeline Engine

Build a tiny declarative pipeline engine in JavaScript: tokenize and parse a Jenkinsfile-like text, run stages, parallel branches, retry, environment and post actions.

The best way to understand a tool is to build a small version of it. In this project you will write a mini pipeline engine: a program that reads text that looks like a declarative Jenkinsfile, understands it, and "runs" it, printing a log with stages, parallel branches, retries, environment variables and post actions. About 150 lines of JavaScript are enough.

Be clear about what this is. The real Jenkins parses Groovy, runs steps on agents, and survives restarts. Our engine does none of that: it understands a tiny subset of the syntax, and its "shell" is a fake that only pretends to run commands. What you will learn is the shape of the problem: text becomes a tree, a tree becomes a plan, and the plan produces a log and a result. That shape is the same inside the real thing.

Three layers

Tokenizer Cuts the text into words, strings and braces.
Parser Turns tokens into a tree of blocks.
Runner Walks the tree and writes the log.

Each layer only knows about the one before it.

1 / 2

The language we accept

Declarative pipelines are nested blocks. A block has a name, optional arguments and optional { ... } contents. That one idea covers almost everything:

GROOVY
pipeline {
  agent any
  environment {
    APP = 'shop'
  }
  stages {
    stage('Build') {
      steps {
        echo "building ${APP}"
        sh 'make build'
      }
    }
  }
}

Look at stage('Build') { ... }, echo "...", agent any and retry(3) { ... }. Each is name, arguments, optional block. Arguments may be written with or without parentheses. The only other form is an assignment such as APP = 'shop' inside environment. So the grammar is small enough to parse by hand.

Step 1: tokenizer and parser

The tokenizer walks the characters and produces small objects: strings (remembering which quote was used, because only double-quoted strings interpolate ${VAR} in Groovy), numbers, identifiers, braces, parentheses, commas, colons, equals signs and newlines. Newlines matter because echo 'a' and echo 'b' on two lines are two statements.

The parser is a recursive descent parser: statement() reads a name, then arguments until it hits a newline, brace or parenthesis, then an optional block, and block() calls statement() until it sees }. The two functions call each other, which is how nesting works.

JS Tokenizer and parser: text to a tree

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The printed tree is the whole pipeline with the punctuation gone. retry has the argument 3 and a block holding one sh step. Everything the runner needs is now a plain object, so it never has to look at text again.

Real Jenkins does not hand-parse like this. Declarative pipelines are parsed with a Groovy-aware grammar, then validated against a model of allowed sections. Our parser accepts any block names; the runner is what decides which ones mean something.

Step 2: the runner

The runner walks the tree. Here are the design decisions, each of which mirrors something in real Jenkins:

  • Environment. The environment block fills a map. Double-quoted strings replace ${NAME} from that map; single-quoted strings are left alone.
  • A virtual clock. Every sh costs one second and sleep n costs n. No real waiting happens, so the log is instant and deterministic.
  • Stages stop on failure. A step that throws marks the build FAILURE and no later stage runs.
  • Parallel. Each branch starts at the same clock time. Afterwards the clock moves to the end of the slowest branch, which is exactly why parallel stages save wall-clock time.
  • Retry. retry(n) runs its block up to n times and rethrows only if the last attempt fails.
  • when. We support one condition, branch 'name', and a failing condition skips the stage. Skipping is not failing.
  • Post. After the stages, always runs every time, success only when the result is SUCCESS, and failure only when it is FAILURE.

The fake shell has two tricks so you can see the features work: exit 1 fails, and any command starting with flaky fails the first two times it is run and passes on the third.

Try it. The run below uses the branch feature/login, so the Deploy stage is skipped. Then change branch to "main" and run again: Deploy executes exit 1, the build fails, Never is not reached, and the failure post action fires.

JS The complete mini engine

Press Run (or Ctrl + Enter) to see the console output here.

No output.

With feature/login, Unit takes 1 + 3 = 4 virtual seconds and Lint takes three runs of one second each (it fails twice, then passes), so 3 seconds. Because they run in parallel the stage costs 4 seconds, not 7. That is the whole argument for the parallel block.

Ideas to extend it

You now have a working base. Each of these is a good exercise, and each maps to a real Jenkins feature from an earlier lesson:

  • Add timeout(time: 5) { ... } as a step that fails if the block's virtual time exceeds the limit (see Common Steps).
  • Add when { not { branch 'main' } } and anyOf/allOf (see Conditions and Post Actions).
  • Add unstable as a result and a post { unstable { ... } } section.
  • Add parameters and a params.NAME lookup (see Environment Variables and Parameters).
  • Make failFast real: when one parallel branch fails, mark the remaining branches as aborted.

When a feature is hard to add, it usually shows why Jenkins is shaped the way it is. For example, real parallel branches are concurrent, so their log lines interleave. Our runner runs them one after another and fakes the timing, which is simpler and good enough for a model.

What is real and what is the stand-in

The syntax you wrote (pipeline, stages, stage, steps, parallel, post, environment, retry, sleep, echo, sh) is real declarative Jenkins vocabulary. The engine is not: it has no Groovy, no agents, no workspace, no real shell and no persistence. Treat it as a diagram you can execute.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the tokenizer remember which quote character a string used?

    In Groovy, double-quoted strings are interpolated and single-quoted ones are literal. The engine copies that rule.
  2. 2. Two parallel branches take 4 and 3 seconds. How long does the parallel stage take?

    Branches start together, so the stage ends when the slowest one ends: 4 seconds.
  3. 3. What happens to later stages after a step throws an error in our engine?

    The loop breaks on the first failing stage. Post conditions are evaluated afterwards with the final result.
  4. 4. What does retry(3) do when all three attempts fail?

    retry only hides failures that eventually pass. If the final attempt fails, the error propagates.

Summary

  • A pipeline engine is three layers: tokenizer, parser (text to a tree) and runner (tree to a log and a result).
  • Declarative syntax is name, arguments and an optional block, nested; one recursive function pair parses it.
  • Parallel time is the slowest branch; retry rethrows only after the last attempt; post conditions run against the final result.
  • The syntax is real, but the engine is a model: no Groovy, agents or real shell.

Finish the course with Common Mistakes and Anti-Patterns.