Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 36 ยท Architecture and Lifecycle

The Data Engineering Lifecycle

Follow data through its lifecycle: generation, ingestion, storage, transformation and serving, plus the undercurrents of security, orchestration and cost.

A data platform can look like a tangle of tools. A lifecycle model gives you a map that does not change when the tools do. It splits the work into stages that data flows through and "undercurrents" that apply to every stage. When someone proposes a new tool, you can ask: which stage does it serve, and what does it do to the undercurrents?

The five stages

Data moves through five stages

Generation Apps and devices create data.
Ingestion Data enters the platform.
Storage Kept durably and cheaply.
Transformation Cleaned, joined, modeled.

Serving closes the loop.

1 / 2
  1. Generation. Source systems create data: application databases, IoT devices, mobile apps, third-party SaaS. Data engineers usually do not own sources, but they must understand them: how often does the schema change, how large is it, what are its guarantees?
  2. Ingestion. Getting data from sources into the platform, in batches or as a stream. The next two chapters cover this in depth.
  3. Storage. Where data rests: a warehouse, a data lake of files, a lakehouse that adds table features to files. Storage choices shape cost, speed and flexibility. Chapter 5 covers formats and lakehouses.
  4. Transformation. Turning raw data into something useful: cleaning, deduplicating, joining, aggregating, modeling. Chapter 6 covers modeling.
  5. Serving. Delivering to consumers: BI dashboards, data science notebooks, ML features, applications, and reverse ETL (pushing warehouse data back into tools like a CRM).

The undercurrents

Six concerns cut across every stage. They are not steps; they are qualities of the whole system.

Undercurrent The question it asks
Security Who can see and change what?
Data management Is the data defined, documented and trustworthy?
DataOps Do we test, deploy and monitor data code like software?
Data architecture Does the overall design fit the needs and change well?
Orchestration What runs when, and what happens on failure?
Software engineering Is the code clean, versioned and tested?

Cost cuts across everything too, and is its own chapter later (Governance and Cost).

The lifecycle as code

To make the stages concrete, here is a small pipeline in which each stage is a function and a log records what happens. This is the shape real orchestrators schedule.

Python Five stages, five functions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that the raw table is stored before transformation. Keeping raw data lets you rebuild derived tables whenever the logic changes. This idea returns in the medallion pattern.

Working backwards from the end

Beginners start at ingestion because it is concrete. Experienced engineers start at serving: what decision will this data support, how fresh must it be, and who owns the definition of each metric? Requirements at the end determine choices at every earlier stage. A dashboard refreshed once a day needs a batch job; a fraud alert needs a stream.

A small habit makes this explicit: write a one-line "data contract" per output.

Python Describe a data product as data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The freshness requirement ("max age") directly selects batch or streaming. You will see that decision again in streaming fundamentals.

Common lifecycle mistakes

  • Skipping the raw layer. Without raw data you cannot reprocess after a bug.
  • Transforming at ingestion. Mixing extraction and business logic makes both hard to change.
  • Ignoring undercurrents until late. Retrofitting security or monitoring is far harder than building it in.
  • No owner. Each dataset needs a named owner, or nobody fixes it when it breaks.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which is NOT one of the five lifecycle stages?

    The stages are generation, ingestion, storage, transformation and serving. Marketing is not a stage.
  2. 2. What is reverse ETL?

    Reverse ETL syncs modeled data from the warehouse into tools such as a CRM or support system.
  3. 3. Why store raw data before transforming it?

    A raw layer lets you reprocess history when you find a bug or need a new metric.
  4. 4. A fraud alert that must fire within seconds most likely needs which approach?

    Seconds-level freshness requires processing events as they arrive.
  5. 5. Which is an undercurrent rather than a stage?

    Security applies to all stages. The others are stages the data moves through.

Summary

  • The lifecycle has five stages: generation, ingestion, storage, transformation and serving.
  • Security, data management, DataOps, architecture, orchestration and software engineering run through all stages.
  • Start from the consumer's freshness and definition needs and work backwards.
  • Keep raw data, separate extraction from business logic, and give every dataset an owner.

Next: OLTP vs OLAP