Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 36 ยท Architecture and Metastore

Execution Engines: MapReduce, Tez and Spark

Compare the engines Hive can use to run queries: MapReduce, Apache Tez, Hive on Spark and LLAP. Learn how map and reduce stages work and how to choose and set the engine.

Hive plans queries, but something else has to run them. That something is the execution engine. Hive has used three over its life, and the change from the first to the second is why a modern Hive feels so much faster than the one in old tutorials. This lesson explains how a distributed query actually runs, then compares the engines.

Engines run on a cluster, so Hive settings are shown read-only. The runnable editors are SQLite stand-ins that simulate the map and reduce idea with plain SQL; they are labelled and do not use any cluster.

The map and reduce idea

Every engine builds on the same pattern: split the data, work on the pieces in parallel, redistribute by key, then combine.

flowchart LR
    I1["split 1"] --> M1["map task:<br/>scan, filter,<br/>partial count"]
    I2["split 2"] --> M2["map task"]
    I3["split 3"] --> M3["map task"]
    M1 -->|shuffle by key| R1["reduce task:<br/>combine for keys A-M"]
    M2 -->|shuffle by key| R1
    M3 -->|shuffle by key| R2["reduce task:<br/>combine for keys N-Z"]
    M2 --> R2
    M1 --> R2
    M3 --> R1
    R1 --> O["output"]
    R2 --> O

Map tasks work where the data is; the shuffle sends each key to one reducer.

For SELECT genre, COUNT(*) FROM books GROUP BY genre, each map task reads a split of the files and counts genres locally. The shuffle sends all partial counts for the same genre to the same reducer, which adds them. The stand-in below mimics this with two SQL steps: a count per block of rows, then a final sum.

SQL Stand-in: partial counts, then a combine step (SQLite, not Hive) ยท database: bookstore

Press Run (or Ctrl + Enter) to see the result here.

The final numbers equal a direct GROUP BY; the engine just does it in two parallel phases. Hive's real map side also uses map-side aggregation (hive.map.aggr=true) to shrink data before the shuffle.

MapReduce

The original engine. A query becomes a chain of MapReduce jobs, each one reading from and writing to HDFS. Strengths: very robust, scales to huge data, minimal memory needs. Weaknesses: every stage writes intermediate data to disk, each job pays a start-up cost of many seconds, and a query with five stages runs five jobs back to back. It is deprecated since Hive 2 and removed from the default in Hive 3. You may still meet it on old clusters.

Apache Tez

Tez keeps the map/reduce vocabulary but represents the whole query as one directed acyclic graph (DAG) of vertices (called Map 1, Reducer 2, ...). Intermediate results flow between vertices through memory and local disk instead of HDFS, and containers are reused, so start-up cost is paid once. For typical queries Tez is several times faster than MapReduce, and it is the default engine in Hive 3 and 4.

Tez features you will see in plans:

  • Vectorization: processing 1,024 rows at a time instead of one, which is much faster on columnar formats such as ORC.
  • Dynamic partition pruning, map joins and bucket-aware joins.
  • Auto-reducer parallelism: the number of reducers is adjusted at run time from real data sizes.

Hive on Spark

Some distributions allow Spark as the engine: SET hive.execution.engine=spark. Spark keeps data in memory between stages and has a flexible DAG scheduler. The setup is more complex (a version of Spark has to match the Hive build), and support varies. Do not confuse it with Spark SQL, which is a separate engine that can read the Hive metastore but does not use HiveServer2 or Hive's optimizer.

LLAP

Live Long and Process adds a layer of persistent daemons on top of Tez. They keep data caches, JIT-compiled code and execution slots warm, so sub-second to few-second queries become possible. With LLAP enabled, short queries run inside the daemons and heavy ones can still use ordinary Tez containers. It is the choice for dashboards and BI tools on Hive.

Comparison

MapReduce Tez Spark LLAP
Plan shape Chain of jobs One DAG One DAG Tez DAG in daemons
Intermediate data HDFS Memory / local disk Memory / local disk Memory, cached
Start-up Slow Moderate Moderate Fast
Typical use Legacy Default, ETL and reports Where Spark is standard Interactive BI
Status Deprecated Recommended Optional Optional

Choosing the engine

The engine is a property, set per session or in hive-site.xml:

SQL
SET hive.execution.engine=tez;      -- recommended
SET hive.execution.engine=mr;       -- legacy
SET hive.execution.engine;          -- show current value

Related properties you will tune later: hive.vectorized.execution.enabled, hive.exec.reducers.bytes.per.reducer, tez.am.resource.memory.mb and hive.tez.container.size (container memory in MB).

Why this matters for you

  • A query that is slow only the first time may be paying container start-up costs.
  • A plan with many reducers and sorts has many shuffles; the engine cannot fix a plan that shuffles too much.
  • Switching engine does not change SQL results, only speed. If results differ, there is a bug or a settings issue, not an engine "feature".
  • File formats and engine work together: ORC with Tez and vectorization is the classic fast combination (see the file formats chapter).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the shuffle do?

    The shuffle moves partial results across the network by key, ready for the combine step.
  2. 2. Why is Tez usually faster than MapReduce for Hive?

    MapReduce writes every stage to HDFS and starts new jobs; Tez avoids both costs.
  3. 3. Which property chooses the engine?

    hive.execution.engine takes values like tez, mr and spark.
  4. 4. What does LLAP add?

    LLAP daemons keep data and code warm, removing start-up cost for short queries.

Summary

  • Engines run Hive's plan in parallel: map tasks work on splits, a shuffle regroups by key, reducers combine.
  • MapReduce is legacy and disk-heavy; Tez (default) runs the whole query as one DAG; Spark is an option; LLAP adds persistent caching daemons.
  • Set the engine with hive.execution.engine; results are the same, only speed changes.
  • Vectorization and columnar formats make Tez even faster.

Next: Databases and DDL