Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 47 ยท MapReduce

Writing a MapReduce Job in Java

Write a complete Hadoop MapReduce job in Java: Mapper, Reducer, Writable types, the Job driver, input and output formats and running it with hadoop jar.

Java is Hadoop's native language, and the Java API is what every other interface (Streaming, Hive, Pig) ultimately sits on. A MapReduce job in Java has three parts: a Mapper class, a Reducer class and a driver (a main method that configures a Job and submits it). You package them into a JAR and run it with hadoop jar.

This lesson writes the classic word count in full, explains each type and setting, and then mirrors the same structure in a runnable Python stand-in so you can watch the lifecycle (setup, map, cleanup, reduce) happen. It assumes you know the ideas from The MapReduce Programming Model and basic Java syntax.

Hadoop and a cluster are not available on this site, so the Java code and the shell commands below are read-only. The runnable examples are Python stand-ins with the same job structure. They are not Hadoop itself.

Writable: Hadoop's serialization types

Keys and values travel between machines and are written to disk, so Hadoop does not use plain String and int. It uses Writable types, which know how to serialize themselves compactly and quickly. The common ones:

Java type Writable
int IntWritable
long LongWritable
double DoubleWritable
String Text (UTF-8 bytes)
boolean BooleanWritable
nothing NullWritable

Keys must also be comparable (WritableComparable) because the shuffle sorts them. Writables are mutable and reused: a good mapper creates its Text and IntWritable once and calls set(...) for each record, instead of allocating millions of objects. Never store a reference to a key or value that the framework passed in, because its contents change on the next call.

The Mapper

Java
import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

public class WordCountMapper
        extends Mapper<LongWritable, Text, Text, IntWritable> {

    private static final IntWritable ONE = new IntWritable(1);
    private final Text word = new Text();

    @Override
    protected void map(LongWritable offset, Text line, Context context)
            throws IOException, InterruptedException {
        StringTokenizer tokens = new StringTokenizer(line.toString());
        while (tokens.hasMoreTokens()) {
            word.set(tokens.nextToken().toLowerCase());
            context.write(word, ONE);
        }
    }
}

The four type parameters are Mapper<KEYIN, VALUEIN, KEYOUT, VALUEOUT>. With the default text input, the key is the byte offset of the line in the file and the value is the line itself. The mapper emits (word, 1) through context.write. The Context object is also how you read configuration, update counters and report progress.

A mapper has two more hooks: setup(Context) runs once before the first record (open a lookup file, read a parameter) and cleanup(Context) runs once after the last (emit a buffered summary, as in the top-N pattern from MapReduce Design Patterns).

The Reducer

Java
import java.io.IOException;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

public class WordCountReducer
        extends Reducer<Text, IntWritable, Text, IntWritable> {

    private final IntWritable total = new IntWritable();

    @Override
    protected void reduce(Text word, Iterable<IntWritable> counts, Context context)
            throws IOException, InterruptedException {
        int sum = 0;
        for (IntWritable count : counts) {
            sum += count.get();
        }
        total.set(sum);
        context.write(word, total);
    }
}

reduce is called once per distinct key, with an iterator over all of that key's values. The iterator is a stream: it reads from the merged shuffle data and cannot be rewound, so you get one pass (copy values into a list only if they are few). The reducer's input types must equal the mapper's output types.

The driver

Java
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount {
    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        Job job = Job.getInstance(conf, "word count");

        job.setJarByClass(WordCount.class);
        job.setMapperClass(WordCountMapper.class);
        job.setCombinerClass(WordCountReducer.class);   // same logic, map side
        job.setReducerClass(WordCountReducer.class);

        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        job.setNumReduceTasks(2);

        FileInputFormat.addInputPath(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));

        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }
}

The driver describes the job; it does not process data. setJarByClass tells Hadoop which JAR to ship to the cluster. Reusing the reducer as the combiner is valid here because summing is associative (see Inside the Shuffle and Sort). waitForCompletion(true) submits the job to YARN (see YARN: Resources and Scheduling) and prints progress.

Anatomy of a Java job

Mapper Turns each record into key-value pairs.
Reducer Folds all values of a key into a result.
Driver Configures and submits the Job.
JAR Everything is shipped to the cluster.
1 / 2

Input and output formats

An InputFormat does two jobs: it divides the input into splits (one map task each) and it supplies a RecordReader that turns a split into (key, value) records.

InputFormat Key Value Notes
TextInputFormat (default) byte offset (LongWritable) the line (Text) one record per line
KeyValueTextInputFormat text before the first tab text after it for tab-separated records
NLineInputFormat offset line each split has N lines
SequenceFileInputFormat as stored as stored Hadoop's binary format

A split normally matches an HDFS block (128 MB), so each map task reads local data. The reader handles lines that straddle a split boundary: a reader skips the partial first line (the previous split's reader finishes it) and reads past its end to complete the last one.

OutputFormat is the mirror image: TextOutputFormat writes key<TAB>value lines, SequenceFileOutputFormat writes binary files, and NullOutputFormat discards output (for jobs that only have side effects). Output goes to a directory with one file per reducer and a _SUCCESS marker. The directory must not exist; otherwise the job fails immediately, so a job never overwrites earlier results by accident.

How many mappers and reducers?

Mappers are not set directly. The number of map tasks equals the number of input splits, which the InputFormat computes. For file-based formats the split size is:

Text
splitSize = max(minSize, min(maxSize, blockSize))

with minSize from mapreduce.input.fileinputformat.split.minsize (default 1) and maxSize from ...split.maxsize (default very large). With the defaults, a split is one HDFS block (128 MB), so a 1 GB file gives 8 map tasks. To get more, smaller maps, lower maxsize; to get fewer, larger maps, raise minsize. Each file is split separately, so 10,000 files of 1 MB give 10,000 maps; that is the small-files problem, and CombineFileInputFormat packs several small files into one split. Non-splittable inputs (such as a gzip-compressed text file) give one split per file whatever the size, because a gzip stream cannot be read from the middle. A good target is a map task that runs for roughly a minute or more: shorter tasks waste time on startup.

Reducers are set explicitly in the driver:

Java
job.setNumReduceTasks(10);       // 10 output files: part-r-00000 ... part-r-00009
job.setNumReduceTasks(0);        // map-only job
job.setNumReduceTasks(1);        // a single globally sorted output file

The default is one reducer, which becomes a bottleneck on any large job. Guidance for choosing a number:

  • Each reducer should process roughly 1 GB or more of map output; thousands of tiny reducers add scheduling and fetch overhead (the number of shuffle transfers is mappers x reducers).
  • A classic rule of thumb is about 0.95 or 1.75 times (nodes x containers per node): with 0.95, all reducers start at once and finish in one wave; with 1.75, faster nodes run a second wave, which balances load better.
  • More reducers means more output files (and more small files downstream), so compact if needed.
  • A single reducer is right only for small results or a total ordering you cannot get any other way. Total ordering across many reducers needs a range partitioner (TotalOrderPartitioner) built from a sample of the keys.
  • Skew matters more than the count: if one key dominates, one reducer does most of the work regardless of how many you configure.
Python Splits and reducer counts

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Custom RecordReaders

The RecordReader turns a split (a byte range) into the (key, value) pairs a mapper sees. TextInputFormat's reader produces (byte offset, line). You write your own when the data has a different record shape, such as a multi-line XML element or a fixed-width binary record. The reader must handle the split boundary rule: it starts at the first record that begins inside its split and reads past the end to finish the last one, so records straddling a boundary are read exactly once. The three methods you implement are initialize (open the split), nextKeyValue (advance to the next record) and getCurrentKey / getCurrentValue, plus getProgress and close. Pair it with a custom InputFormat that returns your reader from createRecordReader, and say so in the driver with job.setInputFormatClass(...). Most of the time, though, an existing format (text, KeyValueTextInputFormat, sequence files, Avro or Parquet) is enough and far less work.

Building and running

Terminal
# compile and package
javac -classpath "$(hadoop classpath)" -d classes WordCount*.java
jar cf wordcount.jar -C classes .

# put input into HDFS, run, and look at the result
hdfs dfs -mkdir -p /user/me/books
hdfs dfs -put moby.txt /user/me/books/
hadoop jar wordcount.jar WordCount /user/me/books /user/me/wc-out

hdfs dfs -ls /user/me/wc-out
hdfs dfs -cat /user/me/wc-out/part-r-* | sort -k2 -nr | head

Generic options such as -D mapreduce.job.reduces=4 can be passed on the command line if the driver uses ToolRunner (the recommended entry point for real jobs). Counters, task logs and the job's progress are shown in the YARN web interface.

A Python stand-in with the same lifecycle

This TryIt mimics the structure above: a Job object collects the mapper, combiner and reducer classes, setup and cleanup hooks run once per task, and Context.write collects output. It is a model of the API shape, not of Hadoop internals.

Python A Job, Mapper and Reducer in Python

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The same program shows why reused objects and one-pass iterators matter in Java: the framework streams data through your classes rather than handing you a whole dataset.

Python Splits, offsets and input records

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each line is produced exactly once even though splits cut the file at arbitrary byte positions, which is the contract of a record reader.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does Hadoop use Text and IntWritable instead of String and int?

    Keys and values cross the network and disk, so they need efficient, framework-controlled serialization. Keys additionally implement comparison for sorting.
  2. 2. With the default TextInputFormat, what are the key and value passed to map()?

    TextInputFormat produces one record per line: the key is the byte offset in the file, the value is the line.
  3. 3. You run the job twice with the same output directory. What happens the second time?

    Hadoop refuses to run if the output path exists, which protects earlier results from being overwritten.
  4. 4. What does the driver class do?

    The driver builds a Job, sets mapper, reducer, types and paths, and submits it. The real work happens in tasks on the cluster.

Summary

  • A Java job = Mapper + Reducer + a driver that configures the Job; it is packaged as a JAR and run with hadoop jar.
  • Keys and values are Writable objects; they are reused, so copy before keeping references.
  • map is called per record, reduce per key; setup and cleanup run once per task.
  • InputFormats produce splits and records; the default reads text lines with byte-offset keys.
  • The output directory must not exist, and each reducer writes one part-r-* file.
  • A combiner can reuse the reducer class when the operation is associative.

Next: Hadoop Streaming with Python.