Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 47 ยท MapReduce

Hadoop Streaming with Python

Write MapReduce jobs in Python with Hadoop Streaming: stdin/stdout mappers and reducers, why reducer input is sorted, and how to test a job with a local pipe.

You do not have to write Java to use Hadoop. Hadoop Streaming lets any program that reads standard input and writes standard output act as a mapper or a reducer. Python, Ruby, Perl, a shell script or a compiled binary all work. The framework runs your program as a child process, pipes records into it and reads records out of it.

This lesson shows the contract, writes a word-count mapper and reducer in Python, and then tests them the way professionals do: with a local pipe (cat | mapper | sort | reducer), which you can reproduce here in a runnable simulation.

A Hadoop cluster is not available on this site, so the hadoop jar command is read-only. The runnable examples run the same mapper and reducer logic through a stand-in that feeds text through simulated stdin and stdout and sorts in between, like the shuffle does.

The streaming contract

  • Input: each line on stdin is one record. For text input, the line is the record itself (no byte offset).
  • Output: each line on stdout is key<TAB>value. The text before the first tab is the key; the rest is the value.
  • Between map and reduce: the framework sorts by key, exactly as in Inside the Shuffle and Sort.
  • Reducer input: the same key<TAB>value lines, sorted by key, but not grouped. A normal reducer receives one long stream, and you notice when the key changes.

That last point is the biggest difference from the Java API. In Java, reduce(key, values) is called per key. In Streaming, your reducer sees all lines of key a, then all lines of key b, and so on. It must keep "the current key" in a variable and emit a result whenever the key changes.

Streaming in four steps

Lines in Records arrive on stdin.
Mapper Prints key, tab, value.
Sort Framework sorts lines by key.
Reducer Watches for the key to change.

Any language that can read and print can join.

1 / 2

The mapper

Python
#!/usr/bin/env python3
# mapper.py
import sys

for line in sys.stdin:
    for word in line.strip().lower().split():
        print(f"{word}\t1")

It reads lines, splits words and prints a pair per word. Nothing Hadoop-specific appears in the code, which is both the appeal and the reason it is easy to test.

The reducer

Python
#!/usr/bin/env python3
# reducer.py
import sys

current_word = None
current_count = 0

for line in sys.stdin:
    word, count = line.rstrip("\n").split("\t", 1)
    count = int(count)
    if word == current_word:
        current_count += count
    else:
        if current_word is not None:
            print(f"{current_word}\t{current_count}")
        current_word = word
        current_count = count

if current_word is not None:               # do not forget the last key
    print(f"{current_word}\t{current_count}")

The reducer depends completely on the sort: it assumes all lines of one word are adjacent. Forgetting the final print after the loop is the classic bug, because the last key never sees a "key changed" event.

Alternatively, itertools.groupby expresses the same logic more compactly:

Python
from itertools import groupby
from operator import itemgetter
import sys

pairs = (line.rstrip("\n").split("\t", 1) for line in sys.stdin)
for word, group in groupby(pairs, key=itemgetter(0)):
    print(f"{word}\t{sum(int(count) for _, count in group)}")

Test locally first

Before submitting anything to a cluster, run the pipeline on your laptop. A pipe with sort in the middle is a faithful model of the whole job:

Terminal
cat input.txt | python3 mapper.py | sort | python3 reducer.py

If this gives the right answer, the Hadoop run will too (only faster, in parallel). Debugging inside a cluster is slow, so a tiny local test is the best habit you can build.

Python The local pipe, simulated

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Try removing the sorted(...) step and run it again: the reducer prints the same word several times with partial counts, because equal keys are no longer adjacent. This is exactly what would happen if the reducer got unsorted input.

Running it on a cluster

With the scripts tested, the job is submitted through the streaming JAR that ships with Hadoop:

Terminal
hadoop jar $HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-*.jar \
  -files mapper.py,reducer.py \
  -mapper "python3 mapper.py" \
  -combiner "python3 reducer.py" \
  -reducer "python3 reducer.py" \
  -input /user/me/books \
  -output /user/me/wc-streaming \
  -D mapreduce.job.reduces=4
  • -files copies the scripts to every node (the distributed cache), so they need not be installed beforehand. Python itself must exist on the worker nodes.
  • -mapper and -reducer are the commands the framework runs. -combiner is optional and works because this reducer is a sum.
  • -input and -output are HDFS paths; as with any MapReduce job, the output directory must not exist.
  • -D sets job properties. Options such as -D stream.num.map.output.key.fields=2 change how many tab-separated fields form the key, which enables composite keys and secondary sort.

Streaming also supports counters and status messages through stderr: print reporter:counter:MyGroup,BadLines,1 to stderr to increment a counter, and reporter:status:parsing to update the task status. Anything else on stderr lands in the task logs, which is where you debug.

Streaming's strengths and costs

Streaming is great for quick data-munging jobs, for teams that live in Python, and for reusing existing tools. The costs:

  • Text overhead. Every record is converted to text, written through a pipe and parsed again. It is slower than native Java by a margin that depends on the job.
  • Hand-written grouping. The reducer does its own key-change tracking.
  • Fewer features. Custom partitioners and input formats need Java classes (they can be passed as options, but they are still Java).
  • Environment. Every node needs the same Python version and libraries.

For analytic work most teams moved on to SQL on Hadoop (Hive) or PySpark (Spark and Beyond), which keep Python but avoid the text-pipe plumbing.

Python Why the reducer must flush the last key

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How does a Streaming mapper send a key and value to Hadoop?

    Streaming communicates through standard streams. The text before the first tab is the key and the remainder is the value.
  2. 2. How does reducer input differ from the Java API?

    Lines for one key are adjacent because of the sort, but the reducer receives one long stream rather than one call per key.
  3. 3. What is the best way to debug a streaming job before using a cluster?

    A local pipe with sort in the middle imitates the job's data flow and finds most bugs in seconds.
  4. 4. Why is the -files option used?

    -files places the listed files in each task's working directory through the distributed cache.

Summary

  • Streaming runs any stdin/stdout program as a mapper or reducer, using tab-separated key<TAB>value lines.
  • Reducer input is sorted by key but not grouped; track the current key and flush the last one.
  • Test with cat | mapper | sort | reducer before using the cluster.
  • Submit with the streaming JAR: -files, -mapper, -reducer, -input, -output.
  • Streaming is convenient but slower than Java because of text conversion; Hive and Spark are the modern alternatives.

Next: Fault Tolerance, Tuning and Limits of MapReduce.