Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 36 ยท Architecture and Lifecycle

Pipeline Architecture Patterns

Compare Lambda, Kappa and medallion (bronze, silver, gold) architectures, learn when each fits, and build a bronze-silver-gold flow in plain Python and SQL.

An architecture pattern is a reusable answer to a recurring question: how do we organize data flows so they stay correct and changeable? Three patterns come up in almost every data platform conversation: Lambda, Kappa and the medallion (bronze, silver, gold) layering. They are not mutually exclusive; the medallion idea in particular sits comfortably on top of the other two.

Lambda architecture

Lambda was designed for the era when batch systems were accurate but slow and streaming systems were fast but approximate. It runs two paths in parallel:

  • A batch layer recomputes everything from the full history, accurately but with delay.
  • A speed layer processes new events immediately, giving fresh but possibly approximate answers.
  • A serving layer merges the two for queries.
flowchart LR
    S["Events"] --> B["Batch layer<br/>(complete, slow)"]
    S --> F["Speed layer<br/>(fast, recent)"]
    B --> V["Serving layer"]
    F --> V

Two code paths, one answer.

The weakness: you maintain the same business logic twice, in two engines, and they must agree. Bugs hide in the differences.

Kappa architecture

Kappa removes the batch layer. Everything is a stream; the log of events is the single source of truth. To reprocess, you replay the log through the new version of the code. One code path, simpler operations, but it needs a stream platform able to retain and replay large histories.

Lambda vs Kappa

Lambda Batch plus speed layer.
Kappa One streaming path.
Lambda cost Logic written twice.
Kappa needs A replayable event log.

Many teams today choose Kappa-like designs.

1 / 2

Modern engines that treat batch as "a bounded stream" blur the line, which is why dual-code Lambda designs are less common now.

The medallion architecture

Medallion organizes data quality layers, usually in a lakehouse:

Layer Also called Contents Rule
Bronze Raw Data exactly as received Never edit; append only
Silver Cleaned Validated, deduplicated, typed, conformed One row per real-world entity or event
Gold Curated Business-ready aggregates and marts Shaped for dashboards and ML

Each layer is rebuilt from the one before it. If a bug is found in the silver logic, you fix the code and rebuild silver and gold from bronze; nothing is lost because bronze is untouched.

A medallion flow you can run

We will push raw events with duplicates and bad rows through bronze, silver and gold, using SQL for the transformations.

Python Bronze, silver and gold in SQL

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Bronze has 5 rows, silver has 3, gold has 2. The duplicate e2 and the unparseable e3 never reach gold, yet they remain visible in bronze for investigation.

Rebuilding from bronze after a bug fix

The real value of the pattern is recoverability. Suppose we later learn that amounts of exactly 5.00 should be excluded as test orders. We change one SQL statement and rebuild:

Python Fix the silver logic and rebuild

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Because the transformation is a function of bronze, rebuilding is safe and repeatable. This is a first taste of idempotent design.

Choosing a pattern

  • Need both accurate history and low latency, and your tools only offer separate batch and stream engines? Lambda is possible, but weigh the double maintenance.
  • Have a replayable event log and mostly event-driven data? Kappa.
  • Mostly batch analytics on a lake or lakehouse? Medallion is almost always a good organizing idea, whichever ingestion style you use.

Do not over-architect. A team with ten tables and a nightly job does not need a three-layer design with a stream. Add structure when pain appears: repeated reprocessing, mixed data quality, or many consumers.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main drawback of Lambda architecture?

    The batch and speed layers duplicate logic, so they can drift apart.
  2. 2. What does Kappa architecture rely on?

    Kappa treats everything as a stream and reprocesses by replaying the log through new code.
  3. 3. In the medallion architecture, which layer holds data exactly as received?

    Bronze is the raw, append-only layer that is never edited.
  4. 4. Why can you safely fix a bug in silver logic?

    Derived layers are functions of bronze; rebuild them after changing the code.

Summary

  • Lambda runs batch and speed layers; Kappa uses one streaming path with a replayable log.
  • Medallion layers data by quality: bronze raw, silver cleaned, gold business-ready.
  • Keep raw data untouched so derived layers can be rebuilt after a bug fix.
  • Add architecture only when real pain justifies it.

Next: Idempotent Pipeline Design