Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 47 ยท MapReduce

MapReduce Worked Examples: Character Count, Weather and Titanic

Three complete MapReduce programs: character count, hot and cold day detection from weather records, and average age by gender, with jars, commands and output inspection.

You learned the model in The MapReduce Programming Model and the Java classes in Writing a MapReduce Job in Java. Word count is the "hello world", but it hides the design choices that real jobs force on you: what should the key be? What goes in the value? Can a combiner help? This lesson works through three small but different programs and shows how to build, run and inspect them.

The three examples are character count (a variation of counting), hot and cold day detection (filtering with a threshold, map-only output) and average age by gender (a group-by with a derived average, where the combiner needs care). Hadoop cannot run on this site, so the Java and shell code is read-only. Each example has a runnable Python stand-in with a tiny MapReduce engine, so you can see the keys, values and output. The sample data is invented for teaching.

A tiny engine to run the examples

All TryIts below use the same helper: run_job(lines, mapper, reducer, combiner=None) maps every input line, groups by key (the shuffle), and reduces each group. It models the data flow, not Hadoop's internals.

Python
def run_job(lines, mapper, reducer, combiner=None):
    shuffled = {}
    for line in lines:
        local = {}
        for key, value in mapper(line):
            local.setdefault(key, []).append(value)
        if combiner:                                   # runs on each mapper's own output
            local = {k: [out for _, out in combiner(k, v)] for k, v in local.items()}
        for key, values in local.items():
            shuffled.setdefault(key, []).extend(values)
    out = []
    for key in sorted(shuffled):                       # keys arrive at the reducer sorted
        out.extend(reducer(key, shuffled[key]))
    return out

Example 1: character count

Task: count how often each character (letter) appears in a text. It is word count with a smaller unit, and it shows the shape of the answer: the key is what you group by (the character), the value is 1, and the reducer sums.

Java
public class CharCountMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
    private static final IntWritable ONE = new IntWritable(1);
    private final Text letter = new Text();

    @Override
    protected void map(LongWritable offset, Text line, Context context)
            throws IOException, InterruptedException {
        for (char c : line.toString().toLowerCase().toCharArray()) {
            if (Character.isLetter(c)) {
                letter.set(String.valueOf(c));
                context.write(letter, ONE);
            }
        }
    }
}

public class SumReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
    @Override
    protected void reduce(Text key, Iterable<IntWritable> values, Context context)
            throws IOException, InterruptedException {
        int sum = 0;
        for (IntWritable v : values) { sum += v.get(); }
        context.write(key, new IntWritable(sum));
    }
}

Notice that this job emits one pair per character, so the map output is as large as the input. A combiner (the same SumReducer, because addition is associative and commutative) shrinks each mapper's output to at most 26 pairs before the shuffle. That is the main lesson here: when the number of distinct keys is small, a combiner removes almost all network traffic.

Python Character count with and without a combiner

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Example 2: hot and cold days from weather records

Task: given daily weather records, list the days that were hot (maximum above 35 degrees Celsius) or cold (minimum below 0). This is a filter job: there is nothing to aggregate, so the reducer can be removed entirely.

Imagine fixed-width or comma-separated lines like 2024-07-14,38.2,24.1 (date, maximum, minimum). The mapper parses a line, tests the thresholds and emits only interesting days:

Java
public class WeatherMapper extends Mapper<LongWritable, Text, Text, Text> {
    @Override
    protected void map(LongWritable offset, Text line, Context context)
            throws IOException, InterruptedException {
        String[] f = line.toString().split(",");
        if (f.length != 3) { return; }                         // skip malformed lines
        String date = f[0];
        double max = Double.parseDouble(f[1]);
        double min = Double.parseDouble(f[2]);
        if (max > 35.0) { context.write(new Text(date), new Text("hot, max " + max)); }
        if (min < 0.0)  { context.write(new Text(date), new Text("cold, min " + min)); }
    }
}

In the driver, job.setNumReduceTasks(0) makes it map-only: the map output is written straight to HDFS, with no shuffle and no sort, so it is the cheapest kind of job (compare MapReduce Design Patterns, the filtering pattern). Each mapper then produces its own part-m-00000 file. If you want a single sorted list, keep one reducer with the default identity reducer instead, and the dates arrive sorted by key.

Real weather files contain bad data such as missing-value markers (for example 9999) and truncated lines. Defensive parsing, with a counter for how many lines were skipped, is part of the job (see context.getCounter(...) in Writing a MapReduce Job in Java).

Python A map-only filter job with a bad-record counter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Example 3: average age by gender (Titanic-style passenger data)

Task: for a passenger list with columns name,gender,age,survived, compute the average age of the passengers who survived, by gender. This is a group-by with an average, and it teaches an important rule: an average is not associative, so you cannot reuse the reducer as a combiner by just averaging averages.

The key is the gender, and the value is the age. The mapper filters (survived == 1), skips rows with a missing age, and emits (gender, age):

Java
protected void map(LongWritable offset, Text line, Context context) throws IOException, InterruptedException {
    String[] f = line.toString().split(",");
    if (f.length < 4 || f[0].equals("name")) { return; }       // header or malformed
    if (!f[3].equals("1") || f[2].isEmpty()) { return; }        // survivors with a known age
    context.write(new Text(f[1]), new DoubleWritable(Double.parseDouble(f[2])));
}

// reducer
double sum = 0; long n = 0;
for (DoubleWritable v : values) { sum += v.get(); n++; }
context.write(key, new DoubleWritable(sum / n));

Why can a combiner not just average? Suppose one mapper sees ages 10 and 20 (average 15) and another sees age 30. The true average is 20, but averaging the two averages gives (15 + 30) / 2 = 22.5, which is wrong because the groups have different sizes. The fix is to pass sufficient statistics instead: the value becomes a pair (sum, count). Both are additive, so the combiner can add pairs and the reducer divides once at the end. This pattern appears whenever you compute means, variances or ratios.

Python Average with a safe (sum, count) combiner

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Building, running and inspecting

For all three programs the workflow is the same (see also Hadoop Streaming with Python for scripts instead of Java):

Terminal
# 1. compile and package into a jar
javac -classpath "$(hadoop classpath)" -d classes *.java
jar -cvf examples.jar -C classes .

# 2. put the input into HDFS
hdfs dfs -mkdir -p /user/ana/weather/input
hdfs dfs -put weather.csv /user/ana/weather/input/

# 3. run (the output directory must NOT exist yet)
hadoop jar examples.jar WeatherDriver /user/ana/weather/input /user/ana/weather/output

# 4. inspect the result
hdfs dfs -ls /user/ana/weather/output           # _SUCCESS and part-m-00000 (or part-r-00000)
hdfs dfs -cat /user/ana/weather/output/part-* | head
yarn logs -applicationId application_1700000000000_0042 | less   # task logs if something failed

After the run, read the counters the framework prints (map input records, map output records, combine input and output records, reduce input groups). In the character-count job, "Combine output records" being tiny compared with "Combine input records" proves the combiner worked. The same program also runs in standalone mode on local files for quick tests (see Installing Hadoop).

Choosing the key and the value

A quick checklist for any new job:

  1. What do I group by? That is the key (character, date, gender).
  2. What must the reducer know about each group? That is the value; if it is a count, send 1; if it is a mean, send (sum, count).
  3. Can any work happen before the reducer? Filter in the map, and add a combiner when the operation is associative and commutative on the values you send.
  4. Do I need a reducer at all? If not, set the reduce tasks to zero.
  5. How do I treat dirty data? Skip and count it, never let one bad line fail the job.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the effect of job.setNumReduceTasks(0)?

    With zero reducers there is no shuffle or sort, so mapper output goes straight to the output directory as part-m files.
  2. 2. Why does averaging the averages in a combiner give a wrong answer?

    An average is not associative. Sums and counts are, so the combiner adds them and the reducer divides once.
  3. 3. In the character count job, why is a combiner so effective?

    With a small key space, summing locally cuts the pairs sent over the network from one per character to at most one per distinct character.
  4. 4. What must be true of the output directory before running hadoop jar?

    Hadoop refuses to overwrite existing output, so a finished result is never lost by accident.

Summary

  • Designing a job starts with the key (what you group by) and the value (what the reducer needs).
  • Character count shows how a combiner collapses a small key space; weather filtering shows a map-only job with defensive parsing and counters.
  • Averages need (sum, count) pairs so that a combiner stays correct.
  • The workflow is compile to a jar, hdfs dfs -put the input, hadoop jar into a fresh output directory, then -cat the part-* files and read the counters.

Next: continue with Hadoop Streaming with Python.