Skip to content
Tutorials Center
My learning

Topics

Lesson 1 of 36 ยท Getting Started

What Is Databricks?

Learn what Databricks is, what problems it solves, who uses it and which pieces (Spark, Delta Lake, Unity Catalog, MLflow) make up the platform.

Databricks is a cloud platform for working with data at scale. You can store large tables, clean and transform them with code, query them with SQL, train machine learning models and schedule all of it to run every night. Everything happens in one web workspace, so analysts, data engineers and data scientists look at the same data instead of passing files around.

The company was founded by the people who created Apache Spark, and Spark is still the engine under most of the platform. Around it you get a table format (Delta Lake), a catalog for permissions (Unity Catalog), an experiment tracker (MLflow), a job scheduler and a SQL interface. This lesson gives you the map. The following lessons zoom into each part.

Honesty note. Databricks is a hosted commercial service, so it cannot run inside this website. Real Databricks code in this course is shown in read-only blocks, and we only show its output when we are certain of it. Every runnable example is a clearly labelled stand-in: a small piece of plain Python (running in your browser) that demonstrates the idea behind the real feature. Stand-ins are never the real product.

The problem Databricks solves

A laptop and pandas are great until the data stops fitting in memory or one machine takes hours. Companies then reach for a cluster: many machines working on slices of the same data. Building and operating that yourself means installing Spark, managing storage, handling security, versioning tables and wiring up schedulers. Databricks packages all of this as a managed service on top of AWS, Azure or Google Cloud.

Three ideas explain most of the design:

  1. Separate storage from compute. Your data lives in cheap cloud object storage. Compute (clusters, warehouses) is rented only while you use it.
  2. Open formats. Tables are stored as Parquet files plus a transaction log (Delta Lake), so other tools can read them too.
  3. One platform, many roles. The same data and governance layer serve BI dashboards, ETL pipelines and ML.

The main building blocks

The Databricks platform

Apache Spark Distributed engine
Delta Lake Reliable tables
Unity Catalog Governance
MLflow Track models

Four pieces you will meet in every chapter

1 / 3
  • Apache Spark splits a big job into many small tasks and runs them in parallel. You write DataFrame code or SQL, and Spark plans the work. See Spark architecture.
  • Delta Lake adds transactions, time travel and updates to files in object storage. See Delta Lake basics.
  • Unity Catalog is the central place for table names and permissions, organised as catalog.schema.table.
  • MLflow records experiments, parameters, metrics and models.
  • Compute comes in several kinds: all-purpose clusters, job clusters, SQL warehouses and serverless compute. See Compute and clusters.

What a typical day looks like

An engineer lands raw files from a source system into cloud storage. A scheduled job reads them, cleans them and writes Delta tables in layers (often called bronze, silver and gold). An analyst queries the gold tables from the SQL editor and builds a dashboard. A data scientist trains a model on the same tables and logs it with MLflow. Permissions come from the catalog, so nobody needs to copy data to share it.

Here is how a first look at the platform usually appears in code. This is real Databricks code, read-only: spark already exists in every notebook, and display renders a table.

Python
df = spark.read.table("samples.nyctaxi.trips")
display(df.limit(5))

The same idea in SQL:

SQL
SELECT pickup_zip, count(*) AS trips
FROM samples.nyctaxi.trips
GROUP BY pickup_zip
ORDER BY trips DESC
LIMIT 5;

A stand-in for the core idea

Why can a cluster be faster than one computer? Because many workers each handle a slice, and a final step combines the results. The stand-in below mimics that with plain Python: it splits a list of trips into partitions, "counts" each partition separately and then merges the partial counts. Real Spark does this across machines; here it is a loop, but the shape of the computation is the same.

Python Stand-in: split, count in parts, merge

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The next stand-in shows the "separate storage from compute" idea. The table is just data that stays put; each "query" is a throwaway function you run against it, and you can run as many as you like without changing the data.

Python Stand-in: storage stays, compute comes and goes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Databricks versus other tools

Databricks is not a database you install; it is a platform that sits on your cloud account. Compared with a classic data warehouse, it handles unstructured files and ML as well as SQL. Compared with running open-source Spark yourself, it saves operations work and adds governance and tooling, at the price of a vendor bill. Many teams use it alongside other tools, since the data is stored in open formats.

Common mistakes

  • Assuming Databricks is Spark. Spark is the engine; Databricks adds the platform.
  • Leaving a cluster running all weekend. Compute is billed while it is on.
  • Copying data into every project instead of sharing tables through the catalog.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which engine runs most Databricks workloads?

    Databricks was founded by the creators of Apache Spark, and Spark remains the main processing engine of the platform.
  2. 2. What does separating storage from compute mean?

    Tables sit in cloud object storage, and clusters or warehouses are started when you need them and stopped afterwards.
  3. 3. Which component is the central place for table permissions?

    Unity Catalog organises tables as catalog.schema.table and controls who may read or change them.
  4. 4. Why are the runnable examples in this course called stand-ins?

    Databricks is a hosted service. The stand-ins are small Python programs that show the concept, and are labelled as such.

Summary

  • Databricks is a managed cloud platform for data engineering, analytics and machine learning.
  • It combines Spark, Delta Lake, Unity Catalog, MLflow, notebooks, jobs and SQL.
  • Storage and compute are separate, and tables use open formats.
  • Real Databricks code here is read-only; runnable examples are Python stand-ins.

Next: The Lakehouse Architecture