Skip to content
Tutorials Center
My learning

Topics

Databricks Tutorial

Learn Databricks step by step: 36 lessons in 9 chapters with examples, quizzes, diagrams and projects.

Start learning →

of lessons completed

Getting Started

  1. 1 What Is Databricks? Learn what Databricks is, what problems it solves, who uses it and which pieces (Spark, Delta Lake, Unity Catalog, MLflow) make up the platform.
  2. 2 The Lakehouse Architecture Understand the lakehouse idea: data lake plus warehouse in one system, and the bronze, silver and gold medallion layers used to organise tables in Databricks.
  3. 3 Getting Started with Databricks Free Edition Create a free Databricks account, understand Free Edition limits versus a paid workspace, and learn how to move around the first screens safely.
  4. 4 Your First Databricks Notebook Write your first Databricks notebook: Python, SQL and Markdown cells, magic commands, display(), the built-in spark session and dbutils helpers.

Platform and Workspace

  1. 5 A Tour of the Databricks Workspace Learn how the Databricks workspace is organised: users, folders, notebooks, the catalog, objects and permissions, plus how account, workspace and metastore relate.
  2. 6 Compute and Clusters Learn the kinds of Databricks compute, how a cluster is made of a driver and workers, autoscaling, auto-termination, runtimes, access modes and cost control.
  3. 7 SQL Warehouses and Serverless Compute Learn what a Databricks SQL warehouse is, how sizes and scaling work, how serverless differs from classic compute, and how the SQL editor, queries and dashboards fit in.
  4. 8 Files, Volumes and Git Folders Learn where files live in Databricks: Unity Catalog volumes, workspace files, legacy DBFS and cloud storage paths, and how Git folders keep your code in version control.

Spark and DataFrames

  1. 9 Spark Architecture: Driver, Executors, Partitions Understand how Apache Spark runs on Databricks: driver and executors, partitions, jobs, stages and tasks, the shuffle, and why skew and small files hurt performance.
  2. 10 DataFrame Basics: Reading, Selecting and Writing Learn the core PySpark DataFrame operations in Databricks: reading tables and files, schemas, select, filter, withColumn, renaming, null handling and writing results.
  3. 11 Transformations, Actions and Lazy Evaluation Learn why Spark is lazy: what transformations and actions are, how the query plan and the Catalyst optimizer work, and when caching and explain() help.
  4. 12 Joins, Aggregations and Window Functions Combine and summarise data in Spark: join types, broadcast joins, groupBy and agg, pivot, and window functions such as row_number, rank, lag and running totals.
  5. 13 Spark SQL, Views and UDFs Mix SQL and Python in Databricks: spark.sql(), temporary views, CTEs, built-in functions, higher-order functions, and when to write a Python or SQL UDF.

Delta Lake

  1. 14 Delta Lake Basics: Creating and Changing Tables Learn Delta Lake, the default table format on Databricks: create managed tables, insert, update and delete rows, see table details and understand why it beats plain Parquet.
  2. 15 The Delta Transaction Log and ACID Look inside a Delta table: the _delta_log folder, JSON commits and checkpoints, how ACID and optimistic concurrency work, and how to read DESCRIBE HISTORY.
  3. 16 Delta Time Travel and RESTORE Query old versions of a Delta table with VERSION AS OF and TIMESTAMP AS OF, compare versions, undo mistakes with RESTORE, and understand how retention and VACUUM limit it.
  4. 17 MERGE INTO and Upserts in Delta Lake Master Delta Lake MERGE INTO: upsert new and changed rows, delete matched rows, handle duplicates in the source, use the Python DeltaTable API and apply slowly changing dimensions.
  5. 18 OPTIMIZE, VACUUM, Clustering and Schema Evolution Keep Delta tables fast and tidy: compact small files with OPTIMIZE, clean up with VACUUM, cluster data with Z-ORDER or liquid clustering, and evolve schemas safely.

Notebooks and Jobs

  1. 19 Notebooks and Magic Commands Learn how Databricks notebooks work: cells, the %sql, %python, %md, %sh and %run magic commands, dbutils helpers, display(), and sharing state between languages.
  2. 20 Widgets and Parameters Make Databricks notebooks reusable with widgets: text, dropdown and multiselect inputs, reading values with dbutils.widgets.get, SQL parameter markers and job parameters.
  3. 21 Jobs and Schedules Run Databricks code on a schedule or on events: create a job, choose task types, set cron or file-arrival triggers, pass parameters, and read run results.
  4. 22 Multi-Task Workflows and Dependencies Build Databricks workflows with several tasks: depends_on, run-if conditions, if/else and for-each tasks, task values, and a runnable dependency-graph stand-in.

Unity Catalog and Governance

  1. 23 The Unity Catalog Namespace Understand Unity Catalog's three-level namespace (catalog.schema.table), the metastore, securable objects, managed versus external tables, and how to organise catalogs.
  2. 24 Grants, Privileges and Groups Control access in Unity Catalog: GRANT and REVOKE, the USE CATALOG and USE SCHEMA gate, SELECT and MODIFY, ownership, groups, service principals and inheritance.
  3. 25 Volumes, Storage Credentials and External Locations Govern files in Databricks: managed and external volumes, /Volumes paths, storage credentials, external locations, and how path-based access is checked.
  4. 26 Lineage, Row Filters and Column Masks Protect sensitive data in Databricks: row filter and column mask functions, dynamic views, tags, data lineage, and audit through system tables, with runnable stand-ins.

MLflow and ML

  1. 27 MLflow Tracking and Experiments Track machine-learning work on Databricks with MLflow: experiments, runs, parameters, metrics, artifacts, autologging, comparing runs, plus a runnable run-tracker stand-in.
  2. 28 The Model Registry in Unity Catalog Register models in Unity Catalog with MLflow: three-level model names, versions, aliases like champion, loading by alias, permissions, and a runnable registry stand-in.
  3. 29 Feature Engineering and Model Serving Build features as governed tables, join them to training data point-in-time, score models in batch with Spark, and serve real-time endpoints with a REST call.

Streaming and Production

  1. 30 Structured Streaming Basics Learn Structured Streaming: unbounded tables, readStream and writeStream, output modes, triggers, checkpoints, watermarks and windows, with runnable Python stand-ins.
  2. 31 Auto Loader: Incremental File Ingestion Ingest new files incrementally with Databricks Auto Loader: cloudFiles format, schema inference and evolution, rescued data, file notification, and a runnable stand-in.
  3. 32 Declarative Pipelines (Delta Live Tables) Define data pipelines declaratively in Databricks: streaming tables, materialized views, expectations for data quality, AUTO CDC, and a runnable dependency stand-in.
  4. 33 Asset Bundles, Testing and CI/CD Ship Databricks work like software: Declarative Automation (Asset) Bundles in YAML, targets for dev and prod, the CLI, unit tests for transforms, and a CI/CD flow.

Projects and Practice

  1. 34 Project: A Medallion Pipeline for Online Orders Build a complete bronze, silver and gold pipeline for online orders: ingestion, cleaning, deduplication, quality checks and daily revenue, with a runnable Python version.
  2. 35 Project: A Governed Sales Mart with a Churn Model Combine governance and ML: design a Unity Catalog sales mart with grants, masks and row filters, train a churn model, register it, and score customers in batch.
  3. 36 Databricks Practice Questions and Exercises Test yourself on Databricks: scenario questions on jobs, Unity Catalog, MLflow, streaming and production, hands-on Python exercises with solutions, and a study checklist.