Skip to content
Tutorials Center
My learning

Topics

Lesson 45 of 47 ยท Projects, Practice and Review

Practice Exercises with Solutions

Twelve hands-on Hadoop and big data exercises on HDFS blocks, replication math, MapReduce jobs, partitioning, skew and file formats, each with a runnable solution.

Reading about HDFS and MapReduce is one thing; doing the arithmetic and writing the jobs yourself is what makes the ideas stick. This lesson gives you twelve small exercises in four groups: block and replication math, MapReduce jobs, partitioning and skew, and file formats. Each group ends with a runnable solution you can edit.

Try every exercise on paper or in your own editor first. Then open the solution, compare, and change the numbers to see what happens.

Hadoop itself cannot run on this site. The solutions are Python stand-ins: a tiny MapReduce runner and plain arithmetic. They follow the same logic as the real framework, but there is no cluster, network or disk behind them. Use 128 MB as the block size unless an exercise says otherwise (see HDFS Architecture).

Four groups of exercises

Blocks Split files and count blocks.
Replication Raw storage and capacity.
MapReduce Write mappers and reducers.
Layout Partitions, skew and formats.

Three exercises per group. Solutions follow each group.

1 / 2

Group 1: blocks and replication

Exercise 1. A 1 GB file (1,024 MB) is stored in HDFS with 128 MB blocks. How many blocks does it have?

Exercise 2. A 700 MB file is stored with 128 MB blocks and replication factor 3. How many blocks are there, how big is the last one, and how many MB of raw disk do the replicas use? (Remember: a block does not reserve a full 128 MB on disk.)

Exercise 3. You must store 10 TB of data with replication factor 3 and keep 25% of the raw disk free for temporary files and growth. Each DataNode has 12 TB of disk. How many DataNodes do you need at least?

Exercise 4. With Reed-Solomon erasure coding RS(6,3) (six data blocks plus three parity blocks), how much raw storage do the same 10 TB need? How does that compare with replication, and how many lost blocks per group can it survive? See Replication and Block Placement.

Solutions for group 1

Python Blocks and replication math

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The reasoning:

  1. 1,024 / 128 = 8 blocks, exactly.
  2. 700 = 5 x 128 + 60, so there are 6 blocks and the last holds only 60 MB. A short block uses only its real size on disk, so the raw usage is 700 x 3 = 2,100 MB, not 6 x 128 x 3.
  3. Replication triples the data to 30 TB. If only 75% of the disk may be used, you need 30 / 0.75 = 40 TB of raw disk, and 40 / 12 is 3.33, so 4 nodes. Real clusters also need enough nodes for rack awareness and failures, so treat this as a floor.
  4. RS(6,3) stores 9 blocks for every 6 of data, an overhead of 1.5x: 15 TB instead of 30 TB. Any 3 of the 9 blocks in a group can be lost without losing data. The price is more CPU and network work when reading after a failure.

Group 2: MapReduce jobs

The solutions use a small helper run_job that does map, optional combine, group by key and reduce, as in The MapReduce Programming Model.

Exercise 5. Input lines look like oslo,-3. Compute the average temperature per city, and make the job safe to use with a combiner. (Hint: an average of averages is wrong.)

Exercise 6. Log lines look like 2026-01-01 ERROR auth timeout (date, level, service, message). Count only ERROR lines per service.

Exercise 7. Find the three most frequent words in a small text. Break ties alphabetically.

Exercise 8. Join users (u1,Ana) with orders (o1,u1,30: order id, user id, amount) using a reduce-side join, and output total spending per user name. Orders from unknown users are dropped. See MapReduce Design Patterns.

Solutions for group 2

Python Four small MapReduce jobs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

What to notice:

  • Exercise 5: the mapper emits (sum, count) pairs, and both the combiner and the reducer add them. A combiner that emitted a partial average would give wrong answers, because averages are not associative. Oslo comes out at -1.33 and Rome at 16.0.
  • Exercise 6: filtering happens in the mapper, so rejected lines never cross the network. Only auth (2 errors) and billing (1) appear.
  • Exercise 7: the top three are be (4), to (4) and or (3). In real Hadoop, global top-N is usually a second job or a reducer that keeps a small heap, because the first job's output is spread over many reducers.
  • Exercise 8: both inputs are tagged U or O so the reducer can tell them apart. Order o4 belongs to u9, who has no name record, so it disappears. Ana spent 42 and Ben 15.

Group 3: partitioning and skew

Exercise 9. A table is stored as one directory per day, /logs/dt=2026-03-01/ through dt=2026-03-31/. A query only needs 10 to 12 March. How many directories must be read?

Exercise 10. A job has one key, hot, with 90 records, and ten other keys with one record each. With 4 reducers and a hash partitioner, one reducer gets all 90 hot records. Add a random-looking suffix ("salt") to hot and compare how the 100 records spread across the reducers. What extra step does salting require? See Inside the Shuffle and Sort.

Solutions for group 3

Python Pruning and salting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Exercise 9 reads 3 of 31 directories, about a tenth of the data. The query engine never opens the other 28, which is the whole point of partition pruning (Partitioning, Bucketing and File Layout).

For exercise 10, the plain run sends every hot record to one reducer, so that reducer does at least 90 of the 100 records. Salting spreads hot over up to four different keys (hot#0 to hot#3), so the busiest reducer's load drops. The price is a second, small aggregation that strips the salt and merges the four partial results. Look at the printed numbers: the busiest reducer falls from 93 records to 48. It is not 25 because hashing four salted keys into four reducers lets two of them collide on the same reducer. More salt values spread the load more evenly.

Group 4: file formats

Exercise 11. A table has 1,000,000 rows and 5 numeric columns of 8 bytes each. A query sums one column. How many MB must a row-oriented file scan, and how many a column-oriented file (ignoring compression and metadata)?

Exercise 12. A sorted country column holds DE, DE, DE, DE, FR, FR, FR, US, US, US, US, US. Compress it with run-length encoding (RLE). How many runs result?

Solutions for group 4

Python Row vs column and RLE

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A row-oriented file must read whole rows, so it scans all 40 MB; a column-oriented file reads just the one column, 8 MB: five times less. The exercise-12 column collapses from 12 values to 3 runs, (DE, 4), (FR, 3) and (US, 5). Sorted or low-cardinality columns compress very well, which is why Parquet and ORC love them (Parquet and ORC Internals).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A 300 MB file is stored with 128 MB blocks. How many blocks does it have?

    300 = 2 x 128 + 44, so there are two full blocks and one 44 MB block: 3 blocks.
  2. 2. Why must a combiner for an average emit (sum, count) rather than a partial average?

    The average of two averages is only correct if both groups have the same size. Sums and counts can be added in any order.
  3. 3. What does salting a hot key do?

    Different salted keys hash to different reducers, so the load spreads. A later step must merge the partial results.
  4. 4. Which layout reads the least data to sum one of five columns?

    A column layout only reads the chunks of the requested column.

Summary

  • Blocks = file size divided by block size, rounded up; a short last block only uses its real size on disk.
  • Raw storage = data x replication factor (or x 1.5 for RS(6,3)), plus free-space headroom.
  • Design aggregations so combiners are safe: emit sums and counts, not averages.
  • Filter in the mapper, tag records for reduce-side joins, and use a second job for global top-N.
  • Partition pruning, salting and columnar layouts are the main levers for less data and balanced work.

Next: Hadoop and Big Data Interview Questions.