Skip to content
Tutorials Center
My learning

Topics

Lesson 3 of 47 ยท Big Data Basics

Hadoop vs a Relational Database

Compare Hadoop with a relational database: schema-on-read vs schema-on-write, scaling, integrity and ACID, workloads and how to choose between them.

A common first question about Hadoop is: "Why not just use MySQL or PostgreSQL?" For most applications, you should. A relational database (RDBMS) is a superb tool for structured data that must be correct, queried quickly and updated all day. Hadoop solves a different problem: storing and processing data that is too large, too varied or too cheap-per-record for a single database server. This lesson compares them on the dimensions that matter and ends with a decision guide.

The comparison is about the storage and processing model, not about brands. Modern systems blur the line: Hive puts SQL on Hadoop, and cloud warehouses scale like Hadoop. Read What Is Big Data? first if the word "scale out" is new. Hadoop cannot run on this site, so the TryIts are small Python stand-ins that contrast the two approaches.

The core difference: when is the structure applied?

An RDBMS uses schema-on-write. You define tables, columns and types first (CREATE TABLE), and the database rejects any row that does not fit. Data is validated and stored in an efficient, indexed layout when it is written, so reads are fast and predictable.

Hadoop uses schema-on-read. You first land raw files in HDFS, whatever they look like: logs, JSON, CSV, images. Structure is applied only when you read the data, by the program or by a table definition such as a Hive external table (see Hive: SQL over Files). Loading is fast and nothing is rejected, but a malformed record shows up later, during processing.

flowchart LR
    subgraph W["Schema-on-write (RDBMS)"]
        A1["Define schema"] --> A2["Validate and store"] --> A3["Fast structured queries"]
    end
    subgraph R["Schema-on-read (Hadoop)"]
        B1["Store raw files"] --> B2["Apply schema when reading"] --> B3["Flexible, slower per query"]
    end

Schema-on-write pays the cost up front; schema-on-read pays it at query time.

Python Schema-on-write versus schema-on-read

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Comparing them point by point

Aspect Relational database Hadoop
Data types structured tables structured, semi-structured and unstructured
Schema schema-on-write, fixed schema-on-read, flexible
Scaling mostly scale up (bigger server); scaling out is hard scale out with commodity machines
Data size gigabytes to a few terabytes comfortably terabytes to petabytes
Access pattern random reads and writes of rows, updates in place large sequential scans, write once, read many
Latency milliseconds per query seconds to hours per job
Integrity ACID transactions, constraints, foreign keys no transactions across files; correctness is the job's problem
Updates row-level UPDATE and DELETE append and rewrite whole files (row-level changes only with table formats such as Hive ACID or Iceberg)
Language SQL Java, Python, SQL via Hive and Spark, and more
Cost per terabyte high (licensed, high-end hardware) low (commodity disks, open source)
Failure handling replicas or backups, mostly manual built-in replication and task retries

Two rows explain most decisions. Scaling: a relational database wants one powerful machine, and when it is full you buy a bigger one at a rapidly rising price; Hadoop adds cheap nodes and gets more storage and more compute together. Integrity: a bank transfer must update two rows atomically, which is what ACID guarantees; a log-analysis job that counts events does not need that guarantee and gains nothing from the cost of providing it.

Note also the access pattern. A database can find one customer by key in microseconds thanks to indexes. Hadoop reads big files from beginning to end; to find one record it still scans a lot of data (partitioning and formats like Parquet shorten that, see Row vs Column Formats). Neither pattern is "better": they serve different questions.

Python Point lookup versus full scan

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When to use which

Choose a relational database when your data is structured and moderate in size, you need transactions, constraints and fast random access, many users query it interactively, or the application updates individual rows (orders, accounts, inventory). Most business applications belong here, and a single well-tuned PostgreSQL or MySQL server goes much further than people expect.

Choose Hadoop (or a data lake built on the same ideas, see Data Lakes, Warehouses and Lakehouses) when the data is huge or growing fast, arrives in many formats, must be kept cheaply for a long time, or needs heavy batch processing such as log analysis, ETL, feature building for machine learning or archives. The two are often used together: the operational database serves the application, nightly extracts (see Getting Data In: Sqoop, Flume, NiFi and Kafka) land in Hadoop, and analytics, reports and models run there. Results then flow back to a database or warehouse for fast serving.

A useful test: if the data fits comfortably on one server and you need transactions, use a database. If you are choosing Hadoop "because it is big data" but your largest table has ten million rows, you probably do not need it.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does schema-on-read mean?

    Hadoop stores files as they are and interprets them at query time, for example through a Hive table definition.
  2. 2. Which task is the better fit for a relational database?

    ACID transactions and constraints are the strengths of an RDBMS. The other tasks are large, append-only or unstructured workloads.
  3. 3. How does Hadoop mainly handle growth in data volume?

    Adding nodes increases both storage and processing capacity.
  4. 4. A team has a 20 GB structured dataset that needs constant row updates and dashboards. What is the sensible choice?

    The data is small and transactional, so a single database handles it better and simpler than a cluster.

Summary

  • An RDBMS validates data on write (schema-on-write) and is built for structured, transactional work with fast random access.
  • Hadoop stores raw files and applies structure on read (schema-on-read), scales out on commodity hardware and excels at large sequential scans.
  • Integrity (ACID), latency and row-level updates favor a database; volume, variety and cost per terabyte favor Hadoop.
  • They usually work together: the database serves the application, Hadoop does the heavy analytics, and results flow back.
  • Do not choose Hadoop for data that fits comfortably on one server.

Next: The Hadoop Ecosystem at a Glance.