Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 36 · Tuning and Integration

Execution Engines – MapReduce, Tez, LLAP and Vectorization

See how Hive runs queries on MapReduce, Tez and LLAP, why Tez DAGs avoid disk writes between stages, and how vectorized execution speeds up scans and aggregates.

HiveQL is only a language. A separate execution engine turns the optimized plan into tasks on the cluster. The engine you use changes latency from minutes to seconds, so it is the first thing to check when Hive feels slow.

None of these engines runs in the browser. Statements are read-only, and the editors use JavaScript models to count the work each design avoids.

MapReduce (legacy)

Hive started on Hadoop MapReduce. A query becomes a chain of MapReduce jobs; between jobs, the intermediate result is written to HDFS and read again. Every job pays startup cost (launching containers), and a query with three joins and an aggregation can become four or five jobs. MapReduce is robust but slow, and it is deprecated: newer Hive versions focus on Tez, so treat MapReduce as a fallback for old clusters.

SQL
SET hive.execution.engine = mr;    -- deprecated; do not choose this on new clusters

Tez

Apache Tez runs the whole query as a single DAG (directed acyclic graph) of vertices. Data flows from vertex to vertex through memory and local disk, not through HDFS, and containers can be reused across tasks and sessions.

SQL
SET hive.execution.engine = tez;   -- the default in Hive 3 and later
flowchart LR
    subgraph MR["MapReduce: 3 jobs, HDFS writes between"]
      direction LR
      M1["Job 1: join"] --> H1[("HDFS")] --> M2["Job 2: group"] --> H2[("HDFS")] --> M3["Job 3: sort"]
    end
    subgraph TEZ["Tez: one DAG"]
      direction LR
      T1["Map vertex: scan and join"] --> T2["Reducer: group"] --> T3["Reducer: sort"]
    end

Same query, fewer stage boundaries and no HDFS round trips.

Tez features that matter for tuning:

  • Container reuse and sessions. hive.server2.tez.initialize.default.sessions keeps a pool of warm Tez sessions, so a query does not wait for a new ApplicationMaster.
  • Dynamic parallelism. Reducer counts are adjusted at runtime from actual intermediate data sizes (auto reducer parallelism).
  • Edges. BROADCAST_EDGE, SIMPLE_EDGE (shuffle), ONE_TO_ONE_EDGE: the vocabulary of the plans in EXPLAIN.
  • Runtime filtering. Hive can build a bloom filter or min/max range from a small join side and push it into the big table scan, so rows that cannot match are dropped before the shuffle (hive.tez.dynamic.semijoin.reduction).

You can count the benefit with a model. The JavaScript stand-in below counts the data written to "HDFS" for the same four-stage query under both designs.

JS HDFS writes: MapReduce versus Tez (model)

Press Run (or Ctrl + Enter) to see the console output here.

No output.

LLAP: Live Long and Process

Even Tez has startup cost per query and re-reads data from HDFS each time. LLAP (Low Latency Analytical Processing, also called Hive Interactive) adds long-lived daemons on the worker nodes. They stay running, so there is no container startup, and they keep an in-memory cache of columnar data (ORC and Parquet, with decoded column chunks and metadata). Hot tables are served from RAM, and repeated dashboards run in seconds.

SQL
SET hive.llap.execution.mode = all;   -- none | map | all | only | auto
  • LLAP is not a different engine: Tez remains the engine, and LLAP daemons execute fragments of the DAG instead of fresh containers.
  • Small, fast fragments run inside the daemons; heavy tasks can still go to regular containers (auto mode).
  • It uses YARN long-running service resources. Size the daemon memory, cache and executors for your cluster; oversizing starves other workloads.
  • The cache works best with ORC; it is shared across users, with security enforced by Hive's authorization layer.

LLAP suits dashboards, BI tools over JDBC, and many concurrent short queries. For rare, huge batch jobs, plain Tez is simpler.

Hive on Spark

Hive can also run on Spark (hive.execution.engine=spark), and many teams use Spark SQL instead with the shared Hive metastore (see integration). Hive on Spark has seen little investment compared with Tez and LLAP, so use it only if your distribution documents and supports it.

Vectorized execution

Classic query engines process one row at a time, calling an operator function per row. Vectorized execution processes a batch of 1,024 rows per call, with each column held as a tight array. This reduces function-call overhead, uses CPU caches and enables SIMD instructions.

SQL
SET hive.vectorized.execution.enabled = true;                  -- default true in recent versions
SET hive.vectorized.execution.reduce.enabled = true;

Vectorization works with ORC and Parquet and with common types and functions. Check coverage with EXPLAIN VECTORIZATION; operators that cannot be vectorized (some UDFs, complex types in some positions) fall back to row mode. This model compares the number of operator calls:

JS Row mode versus batch mode (call counts)

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Choosing

Goal Setting
General batch and ETL Tez with vectorization
Interactive BI, many short queries Tez plus LLAP
Existing Spark pipelines Spark SQL on the same metastore
Old clusters only MapReduce (plan to migrate)

Always verify what you run: SET hive.execution.engine; prints the current value, and the query log shows the Tez application id.

Check your understanding

Quick check · 4 questions

of 4 answered You scored / 4 · perfect!

  1. 1. Why is Tez faster than MapReduce for multi-stage queries?

    MapReduce launches a job per stage and stores each result in HDFS; Tez keeps the query in one DAG with reusable containers.
  2. 2. What does LLAP add on top of Tez?

    LLAP daemons avoid container start-up and cache hot data, giving low-latency queries.
  3. 3. How many rows does a vectorized operator typically process per call?

    Vectorized execution handles a batch of rows as column arrays, cutting per-row overhead.
  4. 4. Which statement shows which operators were vectorized?

    EXPLAIN VECTORIZATION reports vectorized operators and the reasons others were not.

Summary

  • MapReduce is legacy; Tez (a DAG with container reuse) is the standard engine in modern Hive.
  • LLAP adds persistent daemons and a column cache for interactive, concurrent workloads.
  • Vectorized execution processes batches of rows and works best with ORC or Parquet.
  • Check settings with SET hive.execution.engine; and diagnose with EXPLAIN VECTORIZATION.

Next: A Performance Tuning Checklist