Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 47 ยท Data Ingestion and Processing Models

Machine Learning on Hadoop: Mahout and Beyond

Learn what Apache Mahout did for machine learning on Hadoop: recommenders, clustering and classification on MapReduce, why iteration was slow, and what replaced it.

When data is too big for one machine, so is the machine learning on top of it. In the early days of Hadoop, the question "how do I train a model on a terabyte?" had one open-source answer: Apache Mahout. Mahout is a good case study because it shows both what machine learning on MapReduce looked like and exactly why the ecosystem moved on. You may still meet Mahout in older projects, in interviews and in the name of the "recommender on Hadoop" use case.

This lesson explains Mahout's history and its three classic algorithm families (recommenders, clustering, classification), shows why iterative algorithms were painful on MapReduce, and points to what people use today. It builds on Fault Tolerance, Tuning and Limits of MapReduce and Spark, Flink and What Came After MapReduce. Hadoop and Mahout cannot run on this site, so the command lines are read-only; the runnable examples are small Python stand-ins.

A short history

Mahout started in 2008 as a spin-off of the Apache Lucene project, with the goal of implementing scalable machine-learning algorithms on MapReduce. Its name comes from the Hindi word for someone who drives an elephant, a wink at Hadoop's logo. For a few years it was the standard toolbox for large-scale recommenders, clustering and text classification.

Around 2014 the project announced it would stop accepting new MapReduce algorithms because Spark made them obsolete. Mahout reinvented itself as a linear algebra framework with a Scala domain-specific language called Samsara (distributed matrices and vectors with R-like syntax), running on Spark and other engines. The old MapReduce implementations were deprecated and later removed. Today Mahout is a small, niche project, and most teams use Spark MLlib, scikit-learn, XGBoost or deep learning frameworks instead.

The classic algorithm families

Recommenders. Mahout's best-known feature. User-based collaborative filtering finds users whose ratings resemble yours and recommends what they liked. Item-based collaborative filtering finds items rated similarly by the same users ("people who liked this also liked that"). Later versions added matrix factorization with ALS (alternating least squares), which learns hidden factors for users and items. The MapReduce version computed item-item similarity as a pipeline of jobs: build user vectors, count co-occurrences, compute similarities, then produce top-N lists per user. See also the project in Project: Co-purchase Recommendations, which uses the same co-occurrence idea.

Clustering. K-means, fuzzy k-means and canopy clustering group similar points. Canopy was a cheap first pass to choose starting centers.

Classification. Naive Bayes and complementary naive Bayes for text, and random forests, trained on vectors that Mahout built from raw documents (seq2sparse created TF-IDF vectors from text).

A typical command-line flow looked like this (read-only; flags vary by version):

Terminal
mahout seqdirectory -i /data/docs -o /data/docs-seq
mahout seq2sparse -i /data/docs-seq -o /data/docs-vec -wt tfidf
mahout kmeans -i /data/docs-vec/tfidf-vectors -c /data/centers -o /data/clusters -k 20 -x 15 -cl
mahout clusterdump -i /data/clusters/clusters-*-final -d /data/docs-vec/dictionary.file-0 -dt sequencefile

A recommender in miniature

User-based collaborative filtering is easy to understand, and the idea carries over to every library. Represent each user as a vector of ratings, measure cosine similarity between users, and recommend items that similar users rated highly and you have not seen.

Python User-based collaborative filtering

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Why iterative algorithms hurt on MapReduce

Most learning algorithms are iterative: k-means repeats assign-and-update until the centers stop moving; ALS alternates between solving for users and for items; gradient descent loops over the data many times. In MapReduce each iteration is a separate job. That means every pass pays for job startup (scheduling containers and starting JVMs), re-reads the whole input from HDFS, and writes its result back to HDFS (replicated three times) before the next job can start. If k-means needs 20 iterations, the data is read from disk 20 times and written 20 times.

Spark keeps the dataset in memory between passes (cache(), see Caching and Persistence), so only the first pass reads from disk. The difference was commonly around an order of magnitude for iterative jobs, which is why Spark MLlib displaced Mahout's MapReduce algorithms within a few years.

Python Counting the cost of iteration: MapReduce versus in-memory

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

What people use instead

  • Spark MLlib for scalable classification, regression, clustering and ALS recommendations on big DataFrames, with Pipelines (see MLlib Overview and Pipelines, MLlib Clustering and Project: Movie Recommendations with ALS).
  • XGBoost, LightGBM and scikit-learn on a single large machine or with distributed backends (Dask, Spark integration) when data fits in memory after feature building.
  • Deep learning frameworks (PyTorch, TensorFlow) for images, text and audio, with data prepared in Hadoop or a data lake. YARN 3 can schedule GPUs (see Hadoop 2 vs Hadoop 3).
  • Feature stores and managed ML platforms in the cloud for training and serving.

Hadoop's remaining role in ML is mostly storage and data preparation: HDFS or object stores hold the raw data, and Hive or Spark build the training tables. The lesson from Mahout stays useful: choose an engine that keeps data in memory between passes when an algorithm loops.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why were iterative algorithms such as k-means slow on MapReduce?

    Per-iteration job startup and disk I/O dominate. Spark keeps the data cached in memory across iterations.
  2. 2. What did the Mahout project change to after MapReduce fell out of favor?

    The MapReduce algorithms were deprecated; the project refocused on distributed linear algebra.
  3. 3. What does user-based collaborative filtering recommend?

    It finds similar users by comparing rating vectors, for example with cosine similarity, and weights their ratings.
  4. 4. Which is a modern replacement for Mahout's MapReduce algorithms on big tabular data?

    MLlib offers distributed recommenders, clustering and classifiers with in-memory iteration.

Summary

  • Apache Mahout (2008) brought scalable recommenders, clustering and classification to Hadoop MapReduce.
  • Iterative learning algorithms pay job startup plus disk reads and writes on every pass in MapReduce, which made them slow.
  • Mahout deprecated its MapReduce code and moved to a Scala linear algebra DSL on Spark and other engines; Spark MLlib, XGBoost, scikit-learn and deep learning frameworks took over.
  • Collaborative filtering (user-based, item-based, ALS) is the enduring idea; Hadoop now mostly stores and prepares the data.

Next: Practice Exercises with Solutions.