Databricks Tutorial
Learn Databricks step by step: 36 lessons in 9 chapters with examples, quizzes, diagrams and projects.
Start learning →
of lessons completed
Getting Started
- What Is Databricks? Learn what Databricks is, what problems it solves, who uses it and which pieces (Spark, Delta Lake, Unity Catalog, MLflow) make up the platform.
- The Lakehouse Architecture Understand the lakehouse idea: data lake plus warehouse in one system, and the bronze, silver and gold medallion layers used to organise tables in Databricks.
- Getting Started with Databricks Free Edition Create a free Databricks account, understand Free Edition limits versus a paid workspace, and learn how to move around the first screens safely.
- Your First Databricks Notebook Write your first Databricks notebook: Python, SQL and Markdown cells, magic commands, display(), the built-in spark session and dbutils helpers.
Platform and Workspace
- A Tour of the Databricks Workspace Learn how the Databricks workspace is organised: users, folders, notebooks, the catalog, objects and permissions, plus how account, workspace and metastore relate.
- Compute and Clusters Learn the kinds of Databricks compute, how a cluster is made of a driver and workers, autoscaling, auto-termination, runtimes, access modes and cost control.
- SQL Warehouses and Serverless Compute Learn what a Databricks SQL warehouse is, how sizes and scaling work, how serverless differs from classic compute, and how the SQL editor, queries and dashboards fit in.
- Files, Volumes and Git Folders Learn where files live in Databricks: Unity Catalog volumes, workspace files, legacy DBFS and cloud storage paths, and how Git folders keep your code in version control.
Spark and DataFrames
- Spark Architecture: Driver, Executors, Partitions Understand how Apache Spark runs on Databricks: driver and executors, partitions, jobs, stages and tasks, the shuffle, and why skew and small files hurt performance.
- DataFrame Basics: Reading, Selecting and Writing Learn the core PySpark DataFrame operations in Databricks: reading tables and files, schemas, select, filter, withColumn, renaming, null handling and writing results.
- Transformations, Actions and Lazy Evaluation Learn why Spark is lazy: what transformations and actions are, how the query plan and the Catalyst optimizer work, and when caching and explain() help.
- Joins, Aggregations and Window Functions Combine and summarise data in Spark: join types, broadcast joins, groupBy and agg, pivot, and window functions such as row_number, rank, lag and running totals.
- Spark SQL, Views and UDFs Mix SQL and Python in Databricks: spark.sql(), temporary views, CTEs, built-in functions, higher-order functions, and when to write a Python or SQL UDF.
Delta Lake
- Delta Lake Basics: Creating and Changing Tables Learn Delta Lake, the default table format on Databricks: create managed tables, insert, update and delete rows, see table details and understand why it beats plain Parquet.
- The Delta Transaction Log and ACID Look inside a Delta table: the _delta_log folder, JSON commits and checkpoints, how ACID and optimistic concurrency work, and how to read DESCRIBE HISTORY.
- Delta Time Travel and RESTORE Query old versions of a Delta table with VERSION AS OF and TIMESTAMP AS OF, compare versions, undo mistakes with RESTORE, and understand how retention and VACUUM limit it.
- MERGE INTO and Upserts in Delta Lake Master Delta Lake MERGE INTO: upsert new and changed rows, delete matched rows, handle duplicates in the source, use the Python DeltaTable API and apply slowly changing dimensions.
- OPTIMIZE, VACUUM, Clustering and Schema Evolution Keep Delta tables fast and tidy: compact small files with OPTIMIZE, clean up with VACUUM, cluster data with Z-ORDER or liquid clustering, and evolve schemas safely.
Notebooks and Jobs
- Notebooks and Magic Commands Learn how Databricks notebooks work: cells, the %sql, %python, %md, %sh and %run magic commands, dbutils helpers, display(), and sharing state between languages.
- Widgets and Parameters Make Databricks notebooks reusable with widgets: text, dropdown and multiselect inputs, reading values with dbutils.widgets.get, SQL parameter markers and job parameters.
- Jobs and Schedules Run Databricks code on a schedule or on events: create a job, choose task types, set cron or file-arrival triggers, pass parameters, and read run results.
- Multi-Task Workflows and Dependencies Build Databricks workflows with several tasks: depends_on, run-if conditions, if/else and for-each tasks, task values, and a runnable dependency-graph stand-in.
Unity Catalog and Governance
- The Unity Catalog Namespace Understand Unity Catalog's three-level namespace (catalog.schema.table), the metastore, securable objects, managed versus external tables, and how to organise catalogs.
- Grants, Privileges and Groups Control access in Unity Catalog: GRANT and REVOKE, the USE CATALOG and USE SCHEMA gate, SELECT and MODIFY, ownership, groups, service principals and inheritance.
- Volumes, Storage Credentials and External Locations Govern files in Databricks: managed and external volumes, /Volumes paths, storage credentials, external locations, and how path-based access is checked.
- Lineage, Row Filters and Column Masks Protect sensitive data in Databricks: row filter and column mask functions, dynamic views, tags, data lineage, and audit through system tables, with runnable stand-ins.
MLflow and ML
- MLflow Tracking and Experiments Track machine-learning work on Databricks with MLflow: experiments, runs, parameters, metrics, artifacts, autologging, comparing runs, plus a runnable run-tracker stand-in.
- The Model Registry in Unity Catalog Register models in Unity Catalog with MLflow: three-level model names, versions, aliases like champion, loading by alias, permissions, and a runnable registry stand-in.
- Feature Engineering and Model Serving Build features as governed tables, join them to training data point-in-time, score models in batch with Spark, and serve real-time endpoints with a REST call.
Streaming and Production
- Structured Streaming Basics Learn Structured Streaming: unbounded tables, readStream and writeStream, output modes, triggers, checkpoints, watermarks and windows, with runnable Python stand-ins.
- Auto Loader: Incremental File Ingestion Ingest new files incrementally with Databricks Auto Loader: cloudFiles format, schema inference and evolution, rescued data, file notification, and a runnable stand-in.
- Declarative Pipelines (Delta Live Tables) Define data pipelines declaratively in Databricks: streaming tables, materialized views, expectations for data quality, AUTO CDC, and a runnable dependency stand-in.
- Asset Bundles, Testing and CI/CD Ship Databricks work like software: Declarative Automation (Asset) Bundles in YAML, targets for dev and prod, the CLI, unit tests for transforms, and a CI/CD flow.
Projects and Practice
- Project: A Medallion Pipeline for Online Orders Build a complete bronze, silver and gold pipeline for online orders: ingestion, cleaning, deduplication, quality checks and daily revenue, with a runnable Python version.
- Project: A Governed Sales Mart with a Churn Model Combine governance and ML: design a Unity Catalog sales mart with grants, masks and row filters, train a churn model, register it, and score customers in batch.
- Databricks Practice Questions and Exercises Test yourself on Databricks: scenario questions on jobs, Unity Catalog, MLflow, streaming and production, hands-on Python exercises with solutions, and a study checklist.