Skip to content
Tutorials Center
My learning

Topics

Lesson 46 of 47 ยท Projects, Practice and Review

Hadoop and Big Data Interview Questions

Prepare for Hadoop and big data interviews: clear answers on HDFS, YARN, MapReduce, Hive, HBase, file formats and batch vs stream, with small runnable checks.

Interviewers rarely want a recited definition. They want to hear that you know why HDFS uses big blocks, what happens when a node dies, and when you would choose Parquet over Avro. This lesson collects the questions that come up most often, with short answers in plain words and the reasoning behind them. Three small runnable checks let you prove the numbers to yourself instead of memorizing them.

Use it as a review: cover the answer, say yours out loud, then compare. Each answer links to the lesson that explains it in depth.

The real Hadoop stack cannot run on this site. The runnable checks are Python stand-ins for the arithmetic and logic, not a real cluster. Where an answer quotes a default, it is the usual value in recent Hadoop 3 releases; clusters can change them.

Where questions come from

Storage HDFS, formats, HBase.
Compute YARN and MapReduce.
Query Hive and partitioning.
Models Batch vs stream.

Answer with the why, then the trade-off.

1 / 2

HDFS questions

What is HDFS and why is its block so large? HDFS is a distributed file system that splits files into blocks (128 MB by default), stores each block on several DataNodes, and keeps the directory and block map on the NameNode. Blocks are large so that the time to find data (a seek) is tiny compared with the time to read it, and so the NameNode has few blocks to track. See HDFS Architecture.

What does the NameNode store, and what happens if it dies? It stores the namespace (directories, files, permissions) and the file-to-block map in memory, persisted as an fsimage plus an edit log. DataNodes report which blocks they hold. In a single-NameNode cluster, losing it stops the cluster; with High Availability, a standby takes over using shared edits from journal nodes (NameNode HA, Federation and Operations).

How does replication work, and what is rack awareness? Each block is copied to 3 DataNodes by default. The default policy puts the first replica on the writer's node, the second on a different rack and the third on another node of that second rack, so a whole rack can fail without losing data while keeping cross-rack traffic low. If a node dies, the NameNode notices the missing heartbeats and re-replicates its blocks (Replication and Block Placement).

Why are millions of small files a problem? Every file, directory and block is an object in NameNode memory, roughly 150 bytes each. Many tiny files exhaust NameNode memory long before disk space, and they create many tiny map tasks. The fix is to merge files (compaction, SequenceFiles, columnar formats). Check the numbers below.

Python Check: NameNode memory and small files

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Same 100 TB of data, wildly different NameNode memory: tiny files cost the NameNode far more than big ones. That one table is a complete answer to the question.

YARN and MapReduce questions

What is YARN and what are its parts? YARN is the resource manager. The ResourceManager schedules cluster resources, a NodeManager on each machine launches containers (memory plus CPU), and each application has its own ApplicationMaster that asks for containers and tracks its tasks (YARN: Resources and Scheduling). It lets MapReduce, Spark and others share one cluster.

Explain MapReduce in one minute. Mappers read input splits and emit key-value pairs. The framework partitions, sorts and groups the pairs by key (the shuffle), and each reducer receives all values for a key and emits results. Mappers run where the data is stored, which is called data locality (The MapReduce Programming Model).

What is a combiner and when is it safe? A combiner is a mini-reducer that runs on a mapper's output to shrink data before the shuffle. It is safe only if the operation is associative and commutative (sum, count, min, max), and it must emit the same types the reducer expects. An average needs (sum, count) pairs. The framework may run a combiner zero, one or many times (Inside the Shuffle and Sort).

What decides how many mappers and reducers a job has? Mappers: one per input split, usually one per block. Reducers: you choose (mapreduce.job.reduces); too few means slow, skewed reducers, and too many means many small output files and a big mapper-to-reducer fan-in.

What is data skew and how do you fight it? One key (or a few) holds most of the records, so one reducer does most of the work. Fixes: a combiner, salting the hot key and aggregating twice, a custom partitioner, or handling the hot key separately. The check below shows how the partitioner and combiner behave.

Python Check: partitioner, combiner and skew

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With a combiner, 16 records shrink to 3 before the network, and the hot key no longer means a big shuffle. Note that skew in keys can remain even when the data volume shrinks.

Why was MapReduce largely replaced by Spark? MapReduce writes intermediate results to disk between stages and starts a fresh JVM per task, so iterative and interactive work is slow. Spark keeps data in memory across a whole pipeline and plans a DAG lazily (Fault Tolerance, Tuning and Limits, Spark, Flink and What Came After MapReduce). MapReduce's model is still the foundation for understanding all of them.

Hive, HBase and formats

What is Hive? A SQL layer over files in HDFS or object stores. The metastore keeps table definitions; queries are compiled into jobs on Tez (or Spark/MapReduce). It reads data in place (schema-on-read). An external table leaves the files alone when you drop it; a managed table deletes them (Hive: SQL over Files).

Partitioning versus bucketing? Partitioning creates one directory per value of a column (dt=2026-03-10), so queries that filter on it skip whole directories. Bucketing hashes a column into a fixed number of files, which helps joins and sampling. Partition on low-cardinality columns, never on something like user ID (Partitioning, Bucketing and File Layout).

When would you use HBase instead of Hive? HBase is a wide-column store for random reads and writes by row key in milliseconds. Hive is for scans and aggregations over lots of data. HBase writes go to a memstore and a write-ahead log, flush to immutable HFiles and are cleaned up by compaction (an LSM tree). Row-key design decides everything (HBase: Wide-Column Storage).

Row or column format? Parquet, ORC or Avro? Columnar formats (Parquet, ORC) store each column together, so analytic queries read only needed columns, compress well and skip row groups using min/max statistics. Row formats such as Avro suit whole-record writes, streaming and schema evolution (Row vs Column Formats, Avro and Schema Evolution, Parquet and ORC Internals). A format is splittable if a big file can be processed in parallel from the middle; plain gzip text is not.

What is predicate pushdown? Filter conditions are passed down to the reader, which compares them with each row group's min/max statistics and skips groups that cannot match.

Batch vs stream

Batch or streaming, how do you choose? Batch processes bounded data for high throughput and simple reruns; streaming processes unbounded data continuously for low latency. Choose by how fresh the answer must be. Many systems use both (Batch vs Stream Processing).

Event time versus processing time? What is a watermark? Event time is when something happened; processing time is when your system saw it. Events arrive late and out of order, so a watermark is the system's estimate "no events older than this will arrive", used to close windows. Events later than the watermark are dropped or sent to a side output.

Python Check: tumbling windows and late events

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Event 9 arrives after 12, but the watermark (12 - 5 = 7) has not passed the end of window [0, 10), so it still counts. Event 4 arrives after 25 pushed the watermark to 20, so its window is long closed and it is dropped.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why do small files hurt HDFS?

    NameNode memory and task overhead grow with the number of files, not with their size.
  2. 2. When is a combiner safe to use?

    The framework may apply a combiner any number of times in any grouping, so the operation must give the same result regardless.
  3. 3. Which format suits analytic queries that read two columns out of fifty?

    Columnar files read only the needed column chunks and use statistics to skip row groups.
  4. 4. What is a watermark in stream processing?

    Watermarks trade completeness for latency by deciding when an event-time window can be finalized.

Summary

  • HDFS: big blocks, a NameNode with metadata in memory, replicas across racks, re-replication after failures; small files are the classic problem.
  • YARN allocates containers; MapReduce is map, shuffle and reduce with data locality; combiners need associative operations.
  • Hive adds SQL over files; HBase gives random access by row key through an LSM tree.
  • Columnar formats and partitioning cut the data a query reads.
  • Streaming is about event time, windows and watermarks; batch is about bounded data and throughput.

Next: Hadoop and Big Data Cheat Sheet.