Skip to content
Tutorials Center
My learning

Topics

Lesson 3 of 36 ยท Getting Started

Your Data Engineering Toolkit

Set up a practice environment in your browser and learn the daily toolkit: SQL via sqlite3, pandas for transforms, generators for streaming rows and logging.

You do not need a cluster to learn data engineering. Most of the skill is thinking about data movement, and you can practise that with three tools that run anywhere: SQL, Python and pandas. This lesson tours the daily toolkit and shows the patterns you will reuse throughout the course.

Every example runs in your browser. The database is SQLite through Python's built-in sqlite3 module, which behaves like a small, real SQL database. Larger systems add scale and features, but the ideas are the same.

SQL through sqlite3

sqlite3 gives you an in-memory database in one line. Use parameters (?) instead of formatting values into SQL text, which prevents injection bugs and quoting mistakes.

Python A database in memory

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that the query result is a stream of tuples. For large results, iterate rather than calling fetchall(), so memory stays flat.

pandas for in-memory transforms

pandas is the workhorse for data that fits in memory (up to a few gigabytes on a laptop). It reads and writes databases directly, which makes it a handy bridge:

Python SQL to pandas and back

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

If you want a deeper tour, the pandas tutorial covers it fully. For data engineering, the key habit is to keep logic in functions that take a DataFrame and return a DataFrame, so they can be tested.

Generators: process big data in chunks

When a file does not fit in memory, you process it in pieces. Python generators are the natural tool: they produce one item at a time and use constant memory.

Python A streaming pipeline with generators

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each stage pulls from the one before it. Nothing is read until the final loop asks for the next batch, so this design scales to files of any size. Chunked reading is also what pd.read_csv(..., chunksize=n) does.

Logging, not printing

In production nobody is watching a terminal. Use the logging module so messages carry a level, a time and a source, and so they can be sent to a collector.

Python Structured log lines

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Good log lines state counts and names, not just "done". They are the first thing you read when a job fails at 3 a.m.

Beyond the browser: the wider toolkit

You will meet these categories later in the course. Knowing their names now helps you place each new tool:

Category Examples Chapter
Orchestrators Airflow, Dagster, Prefect Orchestration and Quality
Stream platforms Kafka, Pulsar, Kinesis Streaming
Distributed engines Spark, Flink, Trino Ingestion and Batch Processing
Warehouses BigQuery, Snowflake, Redshift Modeling and Warehousing
Table formats Parquet, Delta Lake, Iceberg Storage Formats and Lakehouses
Transformation tools dbt, SQLMesh Modeling and Warehousing

None of them run on this site, so we will teach each idea through a plain-Python stand-in and then show how the real tool expresses it.

A working habit: small, testable functions

Compare a script that does everything in one block with a pipeline of functions. The second is easy to test with a tiny dataset:

Python Test a transform with a tiny dataset

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice df.copy(): a transform that silently changes its input makes bugs hard to find. Returning a new frame is the safer default.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why use ? placeholders in sqlite3 queries instead of building SQL with f-strings?

    Parameters are passed separately from the SQL text, so values are never interpreted as SQL.
  2. 2. What is the main advantage of generators for large files?

    A generator produces items lazily, so only a small amount of data is in memory at once.
  3. 3. Why prefer the logging module over print in a production job?

    Logging gives structured, filterable output that operations tools can collect.
  4. 4. Why does add_revenue return a copy of the DataFrame?

    Pure functions that do not change their inputs are easier to reason about and test.

Summary

  • A practice environment needs only SQL (sqlite3), Python and pandas.
  • Use parameters in SQL, chunks and generators for large data, and logging for visibility.
  • Write transforms as small functions that take and return data, so they can be tested.
  • Bigger platforms add scale; the ideas carry over.

Next: Data Sources, Types and Formats