Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 47 ยท Big Data Basics

Hadoop: History, Role Today and Alternatives

Where Hadoop came from, from Google's papers to Yahoo and the vendors, why cloud object stores replaced much of HDFS, and when Hadoop is still the right choice.

Hadoop is more than twenty years old, which is ancient in software. It started a whole industry, then lost much of its glamour as cloud services appeared. If you are learning it now, you deserve an honest answer to two questions: where did it come from, and is it still worth knowing? This lesson gives both, so that the technical lessons that follow have a context.

You will see the timeline, what the big vendors did, how cloud object stores changed the picture, and a practical checklist for when Hadoop (or its ideas) is still the right choice. It follows The Hadoop Ecosystem at a Glance.

This is a read-only history lesson. The two runnable examples are small Python stand-ins: a timeline you can query and a simple decision helper. They are teaching aids, not real tools, and the decision rules are rules of thumb, not laws.

Where it came from

In the early 2000s, Google had a problem no database could solve: indexing the whole web. Its engineers described their solution in a few research papers:

  • The Google File System (2003): a file system that spreads huge files over thousands of cheap machines and tolerates constant failure. It became the model for HDFS.
  • MapReduce (2004): a simple programming model for processing that data in parallel. It became Hadoop MapReduce.
  • Bigtable (2006): a distributed, wide-column store. It inspired HBase.

At the same time, Doug Cutting and Mike Cafarella were building an open-source web crawler and search engine, Nutch, and ran into the same scaling wall. After reading Google's papers they implemented the ideas in Java. The storage and processing part was split out as its own project in 2006 and named Hadoop, after a toy elephant belonging to Cutting's child, which explains the logo.

From papers to production

Google papers GFS and MapReduce describe the idea.
Nutch An open-source crawler hits the same limits.
Yahoo Runs Hadoop on thousands of nodes.
Open source Becomes an Apache project used worldwide.
1 / 2

Yahoo became the first big adopter and employed Cutting for several years, running Hadoop clusters with thousands of machines. Its work, and that of others such as Facebook and LinkedIn, produced many of the ecosystem projects you will meet: Hive came from Facebook, Pig from Yahoo.

The vendor era

Running Hadoop yourself was difficult, so companies appeared to package it: Cloudera, Hortonworks and MapR sold distributions with installers, management tools and support. Cloudera and Hortonworks merged in 2019, and the MapR assets were later sold. Large cloud providers also offered managed Hadoop (Amazon EMR, Google Dataproc, Azure HDInsight), which runs the same open-source parts without you owning the servers.

An important technical milestone was Hadoop 2 (2013), which introduced YARN. Before it, MapReduce was responsible for both computing and managing resources. After it, other engines such as Spark and Tez could share a cluster with MapReduce. That change is covered in YARN: Resources and Scheduling.

What changed: Spark and the cloud

Two forces reduced Hadoop's dominance, and you should understand both.

1. Spark replaced MapReduce for most new jobs. MapReduce writes intermediate results to disk between every stage, which is slow for iterative work. Spark keeps data in memory where it can and offers friendlier APIs. See Fault Tolerance, Tuning and Limits of MapReduce and Spark, Flink and What Came After MapReduce. Spark often still runs on YARN and reads from HDFS, so it is a replacement for MapReduce, not for Hadoop as a whole.

2. Cloud object stores replaced much of HDFS. Services such as Amazon S3, Google Cloud Storage and Azure Blob Storage store files cheaply, durably and with no servers for you to run. The strongest reason HDFS existed was to put data next to compute. In the cloud you can store data in an object store and start compute clusters only when needed, then shut them down. Storage and compute scale separately, and you pay for compute only while it runs.

HDFS on a cluster Object store in the cloud
Who runs it You (hardware, upgrades) The cloud provider
Data locality Computation runs on the data nodes Data is fetched over the network
Scaling Add nodes (storage and compute together) Storage and compute scale separately
Cost model Pay for servers all the time Pay for storage, plus compute when used
Rename and small-file behaviour Fast, file-system semantics Slower, weaker semantics for some operations

The ideas did not disappear. Table formats such as Iceberg, Delta Lake and Hudi, and engines such as Spark, Trino and Flink, all grew from Hadoop-era thinking, and they commonly run over object stores. Learning HDFS and MapReduce teaches the foundations that these tools still use.

When Hadoop is still the right choice

Hadoop and HDFS on your own machines still make sense in several situations:

  • On-premises data that must not leave your own data centers because of law, security or contracts.
  • Very large, steady workloads where owning hardware is cheaper than renting compute around the clock.
  • Existing investments: a bank or telecom with a mature Hadoop platform, many jobs and trained staff.
  • Data locality at huge scale, where moving petabytes over a network every day would be too slow or costly.

It is usually the wrong choice when your data fits on one machine, when your workload is bursty and a cloud service would sit idle most of the time, or when you have no one to operate a cluster.

Here is the timeline as data, so you can ask questions of it. This is a stand-in written in Python with the dates mentioned in this lesson.

Python Query the Hadoop timeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Now a decision helper. It scores a project against the situations above and prints a suggestion. It is deliberately crude: real decisions weigh cost, skills and risk in more detail.

Python Should this project use Hadoop?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which Google paper inspired HDFS?

    HDFS follows the design of the Google File System. MapReduce inspired Hadoop's compute model and Bigtable inspired HBase.
  2. 2. What did Hadoop 2 introduce that let engines other than MapReduce share a cluster?

    YARN separated resource management from computation, so Spark, Tez and others can run next to MapReduce.
  3. 3. What is a main advantage of cloud object stores over HDFS on your own cluster?

    You keep data in the object store and start compute only when needed, paying for it only while it runs.
  4. 4. Which situation still favours running Hadoop on your own hardware?

    Large steady workloads with on-premises data requirements are the classic case where owned clusters pay off.

Summary

  • Hadoop grew from Google's GFS and MapReduce papers, through Nutch and Yahoo, into an Apache project.
  • Vendors such as Cloudera and Hortonworks packaged it; cloud providers now offer it as a managed service.
  • Hadoop 2 added YARN, which opened the cluster to engines like Spark.
  • Spark replaced MapReduce for most new jobs, and cloud object stores replaced much of HDFS.
  • The ideas live on in modern tools, and on-premises, very large, steady workloads still suit Hadoop.

Next: HDFS Architecture.