Skip to content
Tutorials Center
My learning

Topics

Data Engineering Tutorial

Learn Data Engineering step by step: 36 lessons in 9 chapters with examples, quizzes, diagrams and projects.

Start learning →

of lessons completed

Getting Started

  1. 1 What Is Data Engineering? Learn what data engineering is, what data engineers build, how it differs from data science and analytics, and how a tiny pipeline works end to end.
  2. 2 Data Engineer Roles and Skills See the skills a data engineer needs, how teams are organized, the common job titles, and a practical learning path from SQL and Python to platforms.
  3. 3 Your Data Engineering Toolkit Set up a practice environment in your browser and learn the daily toolkit: SQL via sqlite3, pandas for transforms, generators for streaming rows and logging.
  4. 4 Data Sources, Types and Formats Understand where data comes from, the difference between structured, semi-structured and unstructured data, and how CSV, JSON and logs are parsed in practice.

Architecture and Lifecycle

  1. 5 The Data Engineering Lifecycle Follow data through its lifecycle: generation, ingestion, storage, transformation and serving, plus the undercurrents of security, orchestration and cost.
  2. 6 OLTP vs OLAP Compare transactional OLTP and analytical OLAP systems, understand row versus column storage, and see why analytics should not run on the production database.
  3. 7 ETL vs ELT Learn the difference between ETL and ELT, why cloud warehouses made ELT popular, and how to build both patterns with Python and SQL in your browser.
  4. 8 Pipeline Architecture Patterns Compare Lambda, Kappa and medallion (bronze, silver, gold) architectures, learn when each fits, and build a bronze-silver-gold flow in plain Python and SQL.
  5. 9 Idempotent Pipeline Design Learn why pipelines must be safe to rerun, and how upserts, partition overwrites and deduplication keys make jobs idempotent, with runnable Python examples.

Ingestion and Batch Processing

  1. 10 Ingesting Files: CSV, JSON and Beyond Build a robust file ingestion step: chunked reading, explicit schemas, quarantining bad rows, handling duplicates and late files, with runnable pandas examples.
  2. 11 Ingesting from APIs Pull data from REST APIs safely: pagination, rate limits, retries with backoff, incremental cursors and storing raw responses, using a simulated API in Python.
  3. 12 Incremental Loading and Change Data Capture Compare full and incremental loads, use watermarks, handle deletes with soft deletes and snapshots, and understand how log-based change data capture works.
  4. 13 Batch Transformations Learn the core batch transformation toolkit: cleaning, deduplication, joins, aggregation and window functions, with pandas and SQL examples you can run.
  5. 14 Partitioning and Backfills Learn how partitioning data by date speeds queries and enables safe reruns, and how to run backfills for history using logical dates and idempotent partition jobs.

Streaming

  1. 15 Streaming Fundamentals Understand stream processing: events, append-only logs, topics, partitions, consumer offsets and consumer groups, and when streaming beats batch, using a Python stand-in.
  2. 16 Event Time and Windows Learn the difference between event time and processing time, and compute tumbling, sliding and session windows over a stream with runnable Python code.
  3. 17 Watermarks and Late Data Learn how watermarks track event-time progress, decide when windows close, and how allowed lateness, side outputs and corrections handle late events.
  4. 18 Delivery Guarantees and Deduplication Understand at-most-once, at-least-once and exactly-once delivery, why duplicates happen, and how idempotent sinks and deduplication make streams effectively exactly-once.

Storage Formats and Lakehouses

  1. 19 File Formats: Row-Based vs Columnar Learn how CSV, JSON Lines, Avro, Parquet and ORC store data, why columnar files win for analytics, and how to pick a format for each stage of a pipeline.
  2. 20 Parquet in Practice Open a Parquet file and see inside: row groups, column chunks, min/max statistics, dictionary encoding, compression codecs, filters and schema evolution.
  3. 21 Partitioning, Bucketing and Compaction Learn Hive-style partitioning, partition pruning, how to choose partition columns, the small-files problem and how compaction fixes it, with runnable examples.
  4. 22 Lakehouse Table Formats: Delta, Iceberg, Hudi Understand how Delta Lake, Apache Iceberg and Apache Hudi add transactions, snapshots, time travel and schema evolution to files, with a runnable mini model.

Modeling and Warehousing

  1. 23 Dimensional Modeling and the Star Schema Design a star schema with fact and dimension tables: choose the grain, build surrogate keys, load a fact table and answer business questions with joins.
  2. 24 Slowly Changing Dimensions (SCD Types 1, 2, 3) Handle changing attributes in dimension tables: overwrite with SCD Type 1, keep history with Type 2 effective dates, or store the previous value with Type 3.
  3. 25 Normalization, Denormalization and Data Vault Compare 3NF, star schemas, one big table and Data Vault (hubs, links, satellites), and learn how warehouse layers (raw, staging, marts) combine them.
  4. 26 Warehouse SQL Patterns: ELT, Dedup, Incremental Master the SQL patterns data engineers use daily: ELT with staging tables, latest-record deduplication, MERGE upserts, incremental loads and rebuildable marts.

Orchestration and Quality

  1. 27 Orchestration: DAGs, Schedules and Retries Understand orchestration: tasks and dependencies as a DAG, topological order, schedules, retries with backoff, sensors and failure handling, with a runnable mini scheduler.
  2. 28 Idempotency, Backfills and Reprocessing Make pipelines safe to rerun: idempotent loads with overwrite-by-partition and upserts, deterministic logic, backfilling history, and late-arriving data.
  3. 29 Data Quality: Checks, Contracts and Quarantine Build data quality into pipelines: the dimensions of quality, row-level and dataset-level checks, freshness, anomaly detection, data contracts and quarantining bad rows.
  4. 30 Testing, Monitoring and Observability for Pipelines Test transformations with unit and integration tests, log run metrics, monitor freshness, volume and schema drift, and respond to incidents with alerts and runbooks.

Governance and Cost

  1. 31 Data Catalogs, Metadata and Lineage Learn what a data catalog stores, the types of metadata, ownership and data products, and how lineage graphs answer impact and root-cause questions.
  2. 32 Privacy, PII and Access Control Protect sensitive data in pipelines: classify PII, mask and pseudonymize with hashing, apply least privilege, row and column security, and handle deletion requests.
  3. 33 Cost Optimization and FinOps for Data Platforms Cut data platform spend: understand storage, compute and egress costs, prune partitions, right-size clusters, use lifecycle tiers, and build cost visibility and budgets.

Projects and Practice

  1. 34 Project: A Batch ETL Pipeline End to End Build a complete daily batch pipeline in Python: extract messy CSV, validate and quarantine, transform, load a star schema idempotently, and publish a mart with run metrics.
  2. 35 Project: A Mini Lakehouse with Bronze, Silver and Gold Build a mini lakehouse in Python: land raw JSON, write partitioned Parquet bronze, clean to silver with dedup, aggregate to gold, then compact files and time travel.
  3. 36 Practice Exercises and Interview Questions Hands-on data engineering practice: SQL and pandas exercises (dedup, sessionization, SCD2), design questions, trade-off discussions and interview answers to rehearse.