Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 36 ยท Architecture and Metastore

Hive Architecture

Understand Hive architecture: clients, HiveServer2, the driver, compiler, optimizer, metastore, execution engine and storage, and what each component is responsible for.

To use Hive well you need a clear picture of what happens between pressing Enter on a query and seeing rows. Hive is not one program but a group of cooperating services. This lesson names each part, says what it does, and shows how they connect.

This is a conceptual lesson, so the diagrams and Hive statements are read-only. The one runnable editor is a SQLite stand-in that shows an EXPLAIN plan, the same idea as Hive's own EXPLAIN.

The big picture

flowchart TB
    subgraph Clients
        C1["Beeline"]
        C2["JDBC / ODBC / BI tools"]
        C3["Python, Hue, notebooks"]
    end
    Clients --> HS2["HiveServer2<br/>sessions, auth, Thrift API"]
    subgraph Core["Inside HiveServer2"]
        D["Driver"] --> CP["Compiler / parser"]
        CP --> OP["Optimizer"]
        OP --> EX["Executor"]
    end
    HS2 --> D
    CP <--> MS["Metastore service<br/>(RDBMS)"]
    EX --> ENG["Execution engine<br/>Tez / MapReduce / Spark"]
    ENG --> ST[("Storage<br/>HDFS / S3 / ADLS")]
    ENG --> YN["Resource manager<br/>YARN"]

Queries enter through HiveServer2; the metastore supplies table information; the engine reads storage.

Clients

Clients send HiveQL and show results. They never touch the data directly. Beeline, JDBC/ODBC drivers (used by Tableau, Power BI, DBeaver, Superset), the Hue web UI and Python libraries all speak to HiveServer2 over Thrift or JDBC. The legacy Hive CLI is the exception: it embedded everything in the client, which is why it was replaced.

HiveServer2 (HS2)

HS2 is the front door. It accepts connections, authenticates users (Kerberos, LDAP, or custom), keeps one session per connection with its own settings, and runs many queries at once. It also enforces authorization, such as SQL standard grants or Apache Ranger policies. You scale Hive's concurrency by running several HS2 instances behind ZooKeeper-based service discovery.

The Driver

For each statement, HS2 creates a Driver. It is the conductor: it receives the HiveQL string, asks the compiler for a plan, handles the query's lifecycle and fetches results for the client. The Driver also tracks the transaction a query belongs to, when ACID tables are involved.

Compiler and optimizer

The compiler does three things in order:

  1. Parse the text into an abstract syntax tree (it uses ANTLR).
  2. Semantic analysis: ask the metastore whether tables and columns exist, check types, resolve names, expand views.
  3. Plan: produce a logical plan, a tree of operators such as table scan, filter, join, group-by and select.

The optimizer then rewrites this plan to make it cheaper. Hive uses Apache Calcite, a cost-based optimizer, plus rule-based rewrites such as predicate push-down (filter early) and partition pruning (skip folders that cannot match). The result is a physical plan the engine can run.

You can see a plan with EXPLAIN:

SQL
EXPLAIN
SELECT genre, COUNT(*)
FROM books
GROUP BY genre;

Hive prints a tree of stages and operators. The shape of the output differs from engine to engine, so read it for structure (scan, group, reduce), not for exact text.

Metastore

The metastore holds all metadata: databases, tables, columns and types, partition lists, file formats, table properties, statistics and permissions. It is stored in a relational database and reached through the Hive Metastore Service over Thrift. The next lesson, The Hive Metastore, covers it in depth.

Execution engine

The plan is run by an engine. Hive can use:

  • MapReduce: the original engine, disk-heavy and slow to start; deprecated.
  • Apache Tez: the default in current releases. It builds one directed acyclic graph (DAG) for a whole query rather than a chain of MapReduce jobs, and reuses containers.
  • Apache Spark (Hive on Spark): possible in some distributions.

Engines ask YARN (or Kubernetes) for resources, start tasks near the data and write intermediate results. See Execution Engines: MapReduce, Tez and Spark.

Storage and I/O layer

Data is read through Hadoop's FileSystem API, so the same Hive works on HDFS, Amazon S3 (s3a://), Azure (abfs://) and Google Cloud Storage. Two plug-in types decide how bytes become rows:

  • InputFormat / OutputFormat: how files are split and read (text, ORC, Parquet).
  • SerDe (serializer/deserializer): how one record is turned into columns. Examples: LazySimpleSerDe for delimited text, OpenCSVSerde, JSON SerDes, and the built-in ORC and Parquet SerDes.

When a table declares ROW FORMAT DELIMITED or STORED AS ORC, it is choosing a SerDe and file format.

LLAP: an optional accelerator

LLAP (Live Long and Process) is a set of long-running daemons that cache data and run query fragments in memory. They remove container start-up time, which matters for dashboards that fire many small queries. It is optional; many clusters run Tez only.

Following a query through the parts

For SELECT ... FROM sales WHERE year = 2025: Beeline sends the text to HS2; the Driver gets it; the compiler asks the metastore for the columns of sales and the list of partitions; the optimizer keeps only the year=2025 partition folder; Tez runs scan and aggregate tasks on the cluster, reading the files from storage; the Driver collects the rows and HS2 streams them back. A full trace is in The Life of a Query.

Stand-in: reading a plan

SQLite has its own EXPLAIN QUERY PLAN. It is not Hive's output format, but it teaches the same habit: check whether a query scans everything or uses an index.

SQL Stand-in: EXPLAIN QUERY PLAN (SQLite, not Hive) ยท database: bookstore

Press Run (or Ctrl + Enter) to see the result here.

Hive adds statistics, partition pruning and cluster stages on top, but the question is the same: what will be read, and in what order?

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which component authenticates users and manages sessions?

    HiveServer2 handles connections, authentication, sessions and authorization.
  2. 2. What does the optimizer do?

    Hive's optimizer, built on Calcite, reduces the work the engine has to do.
  3. 3. What is a SerDe?

    SerDe stands for serializer/deserializer.
  4. 4. Which engine builds a single DAG for a query and is the default in modern Hive?

    Tez replaced MapReduce as the default execution engine.

Summary

  • Clients connect to HiveServer2, whose Driver, compiler and optimizer produce a plan.
  • The metastore supplies metadata; the execution engine (Tez by default) runs the plan on the cluster.
  • Data stays in storage (HDFS or object stores) and is read through InputFormats and SerDes.
  • EXPLAIN shows the plan; LLAP is an optional in-memory accelerator.

Next: The Life of a Query