Skip to content
Tutorials Center
My learning

Topics

Lesson 1 of 36 ยท Getting Started

What Is Data Engineering?

Learn what data engineering is, what data engineers build, how it differs from data science and analytics, and how a tiny pipeline works end to end.

Every company that uses data has the same quiet problem: the data is rarely where the people who need it are. Orders live in an application database, clicks in log files, payments in a vendor's API, and a spreadsheet somewhere holds the numbers finance actually trusts. Data engineering is the discipline of collecting that scattered data, moving it, cleaning it, storing it, and delivering it in a shape that analysts, data scientists, dashboards and applications can use reliably.

If a data scientist asks "which customers will churn?", a data engineer makes sure the question can be answered at all: the right events are recorded, joined with the right customer table, refreshed every morning, and checked for mistakes. This course teaches that craft with runnable Python and SQL, so you can try every idea in your browser.

Everything on this page runs for real in your browser: Python (Pyodide) with pandas and the built-in sqlite3 module. Real platforms such as Kafka, Spark, Airflow or a cloud warehouse cannot run on this site, so lessons about them use small, clearly labelled stand-ins written in plain Python. Whenever a stand-in appears, we say so.

What a data engineer actually builds

The core product of a data engineer is the data pipeline: an automated chain of steps that moves data from sources to destinations and transforms it on the way.

A data pipeline in four steps

Ingest Pull data from sources.
Clean Fix types, drop junk.
Model Join and shape it.
Serve Dashboards, ML, apps.

Each step can fail, so each step needs checks.

1 / 2

Beyond pipelines, a data engineer also designs storage (warehouses, data lakes, file formats), keeps the system observable (alerts when a job fails or data looks wrong), controls cost (cloud bills grow fast), and protects governance (who may see which data).

Data engineering, analytics and data science

The three roles overlap, but the center of gravity differs.

Role Main question Typical output
Data engineer How do we get trustworthy data to the right place, on time? Pipelines, tables, platforms
Data analyst What happened, and why? Dashboards, reports, SQL queries
Data scientist What will happen, and what should we do? Models, experiments, forecasts

A useful rule of thumb: analysts and scientists are the consumers of data products; engineers are the producers. A good engineer thinks like a product owner whose users are other people's queries.

A tiny pipeline you can run

Let us build the smallest honest pipeline: raw, slightly messy sales records come in; we clean them; we load them into a SQL table; we query a summary. Each step is a plain Python function so you can see the structure.

Python A four-step pipeline in 40 lines

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice three habits that separate pipelines from one-off scripts:

  1. Rejected rows are kept, not silently dropped. We printed how many failed. Real pipelines write them to a "quarantine" table for inspection.
  2. Cleaning is a function. It can be tested and reused.
  3. The load is explicit. Data moves into a defined schema, not into a variable that disappears.

The three big shifts that made the field

Data engineering existed in the 1990s as "ETL development" and data warehousing. Three changes reshaped it:

  • Volume and variety. Logs, events, JSON from APIs and sensor streams do not fit neatly into a single relational database.
  • The cloud. Storage and compute became cheap, rentable and separable, so teams stopped sizing hardware and started designing for cost.
  • Software engineering practices. Version control, code review, tests and CI now apply to data code too, often called "DataOps".

Good pipelines are boring

A pipeline succeeds when nobody notices it. The properties to aim for are:

  • Reliable: runs on schedule and recovers from failures.
  • Correct: numbers match the source of truth.
  • Idempotent: running twice gives the same result (see idempotent pipeline design).
  • Observable: you learn about a problem before the CEO does.
  • Cheap enough: the compute bill is proportional to the value.

Here is a second small example showing why correctness needs an explicit check: a "row count reconciliation" between source and destination.

Python Reconcile source and destination

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This is the simplest "data quality check". Later chapters build much richer ones.

What you will learn in this course

The nine chapters follow the life of a data platform: getting started, architecture, ingestion and batch processing, streaming, storage formats and lakehouses, modeling and warehousing, orchestration and quality, governance and cost, and finally projects. You can read the track overview at any time.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main product a data engineer builds?

    Data engineers produce pipelines, tables and platforms. Models are the data scientist's product and dashboards are usually the analyst's.
  2. 2. Why did our tiny pipeline keep the rejected rows instead of deleting them?

    Silent data loss is the worst failure mode. Keeping rejected rows makes problems visible and fixable.
  3. 3. Which pair describes consumers and producers of data best?

    Engineers produce reliable data products; analysts and scientists mostly consume them to answer questions and build models.
  4. 4. What did the row-count reconciliation check detect?

    Comparing counts between source and destination reveals missing rows, here the ids divisible by 10.

Summary

  • Data engineering moves, cleans, stores and serves data so others can use it reliably.
  • The central artifact is the data pipeline: ingest, clean, model, serve.
  • Engineers produce data products; analysts and scientists mostly consume them.
  • Good pipelines are reliable, correct, idempotent, observable and cost-aware.
  • Even tiny pipelines should keep rejected rows and verify counts.

Next: Data Engineer Roles and Skills