Skip to content
Tutorials Center
My learning

Topics

Lesson 4 of 47 ยท Big Data Basics

The Hadoop Ecosystem at a Glance

A map of the Hadoop ecosystem: HDFS, YARN, MapReduce, Hive, Pig, HBase, Spark, Kafka and ZooKeeper, what each tool does and how they fit together.

Hadoop is often described as "a tool", but it is really a family. At the center sit three parts that store data, schedule work and compute. Around them live dozens of projects for SQL, streaming, NoSQL, coordination and ingestion. Beginners get lost because the names (Hive, Pig, HBase, Sqoop, Flume, Oozie...) all sound alike.

This lesson is a map. You will learn the three core layers, then the main neighbours and the problem each one solves. Every tool gets its own lesson later, so here we only place it on the map. It follows Distributed Systems in Plain Words.

Nothing in this lesson runs on a cluster. The diagram is a picture, and the runnable example at the end is a small Python stand-in that looks up "which tool solves which problem" so you can practise the map.

The core: storage, scheduling, compute

Hadoop itself has three layers that sit on top of each other:

The three core layers

HDFS Stores huge files across many disks.
YARN Decides which job gets which CPU and memory.
MapReduce The original engine that computes in parallel.

Storage at the bottom, scheduling in the middle, compute on top.

1 / 2
  • HDFS (Hadoop Distributed File System) splits big files into blocks and spreads replicated copies across the cluster. See HDFS Architecture.
  • YARN (Yet Another Resource Negotiator) is the cluster's resource manager. It hands out CPU and memory to applications. See YARN: Resources and Scheduling.
  • MapReduce is the first processing framework: a simple map, shuffle and reduce model. See The MapReduce Programming Model.

The important design idea is to move the computation to the data: instead of pulling terabytes across the network, YARN starts tasks on the nodes that already hold the blocks.

The neighbours

Many other projects use those three layers (or replace one of them). Grouped by job:

SQL and query tools

  • Hive gives you SQL over files in HDFS. You write SELECT ... GROUP BY, and Hive turns it into distributed jobs. (Hive: SQL over Files)
  • Pig offers a dataflow language (Pig Latin) for step-by-step transformations. (Pig Latin Dataflows)

NoSQL storage

Faster compute

Moving data in

  • Kafka is a distributed log for streams of events.
  • Sqoop, Flume and NiFi pull data from databases, log files and many other sources. (Getting Data In)

Coordination

Workflow and operations

How they fit together

A typical pipeline might read like this: events are written to Kafka, a stream job lands them as files in HDFS, Spark or Hive transforms them, results go to HBase for serving or to a dashboard, and ZooKeeper plus YARN keep the machinery running.

flowchart TD
  K["Kafka, Flume, Sqoop<br/>(ingest)"] --> H["HDFS<br/>(storage)"]
  H --> E["MapReduce, Spark, Tez<br/>(compute)"]
  Y["YARN<br/>(resources)"] -.-> E
  E --> Q["Hive, Pig<br/>(query)"]
  E --> B["HBase<br/>(serving)"]
  Z["ZooKeeper<br/>(coordination)"] -.-> H
  Z -.-> B

Data flows left to right and top to bottom; dotted lines are services the others depend on.

Two things are worth remembering. First, the pieces are loosely coupled: you can run Spark on YARN without MapReduce, or Hive on Tez without touching Pig. Second, many modern setups replace HDFS with a cloud object store, which is why Hadoop: History, Role Today and Alternatives is next in the curriculum.

Practise the map

The following stand-in is a small lookup table in Python. It is not a Hadoop API: it simply stores, for each tool, the problem it solves and the layer it belongs to, and lets you ask questions. Run it, then change the needs list to test yourself.

Python Which tool solves which problem?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Now a second stand-in: a tiny planner that, given a job description, suggests the tools you would reach for. Real architecture decisions are subtler, but the exercise builds the habit of asking "which layer is this?".

Python A tiny architecture planner

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the last job: a 10 GB CSV report needs no Hadoop at all. Choosing not to use the ecosystem is often the right call.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which three components form the core of Hadoop?

    HDFS stores data, YARN allocates resources, and MapReduce computes. The others are neighbours that use or extend the core.
  2. 2. Which tool lets analysts run SQL over files stored in HDFS?

    Hive translates SQL queries into distributed jobs that read the files.
  3. 3. What is ZooKeeper mainly used for in a Hadoop setup?

    ZooKeeper holds tiny pieces of shared state. HDFS high availability and HBase depend on it for coordination.
  4. 4. Why does Hadoop try to run tasks on the nodes that hold the data?

    Shipping a small program to the data is far cheaper than shipping terabytes to the program.

Summary

  • Hadoop's core is HDFS (storage), YARN (resources) and MapReduce (compute).
  • Hive and Pig add query languages, HBase adds fast key lookups, Spark adds faster compute.
  • Kafka, Sqoop, Flume and NiFi bring data in; ZooKeeper coordinates; Oozie and Airflow schedule.
  • The parts are loosely coupled, and many can be mixed or replaced.
  • Move the computation to the data, and do not use the ecosystem when one machine is enough.

Next: Hadoop: History, Role Today and Alternatives.