Skip to content
Tutorials Center
My learning

Topics

Lesson 2 of 36 ยท Getting Started

Installing Hive and Run Modes

See what Apache Hive needs (Java, Hadoop, a metastore database), how to install it, which configuration files matter, and the difference between local, embedded and remote modes.

Installing Hive is mostly about assembling its neighbours: a Java runtime, a Hadoop installation (or at least its libraries), a database for the metastore, and a few configuration files. This lesson walks through the pieces, the run modes, and the first-time initialization step that beginners most often forget.

Everything on this page is a read-only walkthrough. Hive cannot run in the browser, so commands are shown for you to try on your own machine, a virtual machine or a container. The runnable editor at the end is a labelled stand-in.

What you need

Component Why Hive needs it
Java (JDK 8 or 11 for Hive 3, JDK 17 for newer releases) Hive is a Java application. Always check the release notes for the supported JDK.
Hadoop (HDFS and YARN) Provides storage and, with MapReduce or Tez, the cluster to run jobs. For a laptop, pseudo-distributed mode is enough.
A metastore database Stores table definitions. Derby is built in for trials; MySQL or PostgreSQL for real use.
The Hive binary release Downloaded as a .tar.gz from the Apache site.

Set the environment variables so the scripts can find everything:

Terminal
export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
export HADOOP_HOME=/opt/hadoop
export HIVE_HOME=/opt/hive
export PATH=$PATH:$HADOOP_HOME/bin:$HIVE_HOME/bin

Create the warehouse folders

Hive keeps managed table data in a warehouse directory in HDFS, by default /user/hive/warehouse. Create it, plus a temporary directory, and make them group-writable:

Terminal
hdfs dfs -mkdir -p /tmp /user/hive/warehouse
hdfs dfs -chmod g+w /tmp /user/hive/warehouse

Initialize the metastore schema

Before the first start, Hive needs its metastore tables created in the backing database. The schematool utility does this:

Terminal
schematool -dbType derby -initSchema

Replace derby with mysql or postgres when you use those databases. If you skip this step, Hive fails at start-up with an error about a missing metastore schema, which is one of the most common first-day problems.

The configuration files

Hive reads its settings from $HIVE_HOME/conf:

  • hive-site.xml: the main file. Metastore connection, warehouse location, execution engine and most tuning options live here.
  • hive-env.sh: environment variables such as heap size and the Hadoop home.
  • hive-log4j2.properties: logging.

A minimal hive-site.xml for a MySQL metastore looks like this:

XML
<configuration>
  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://localhost:3306/metastore_db?createDatabaseIfNotExist=true</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.cj.jdbc.Driver</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hive</value>
  </property>
  <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/user/hive/warehouse</value>
  </property>
</configuration>

The JDBC driver jar must also be copied into $HIVE_HOME/lib.

The three metastore modes

Where the metastore lives and how Hive talks to it is the main deployment choice.

flowchart TB
    subgraph E["Embedded"]
        E1["Hive CLI + metastore code<br/>+ Derby, one JVM"]
    end
    subgraph L["Local"]
        L1["Hive client + metastore code"] --> L2[("MySQL / PostgreSQL")]
    end
    subgraph R["Remote"]
        R1["Hive clients"] --> R2["Metastore service (Thrift)"] --> R3[("MySQL / PostgreSQL")]
    end

Embedded keeps everything in one process; remote puts a metastore service between clients and the database.

  • Embedded: the default. Derby runs inside the same JVM as the client, so only one session at a time can use it. Good for a first experiment, useless for teams.
  • Local: the metastore code still runs inside the client, but the database is an external MySQL or PostgreSQL server. Many users can work, but every client needs the database credentials.
  • Remote: a separate Hive Metastore Service (HMS) listens on port 9083. Clients point to it with hive.metastore.uris. This is the production choice: credentials stay on the service, and Spark or Trino can reuse the same service.

Separately from the metastore, query execution also has modes: local mode runs small jobs in one JVM on the client (useful when testing), while cluster mode submits them to YARN. You will meet the switch hive.exec.mode.local.auto in the tuning chapter.

Starting the services

Once the schema is initialized you start the two services that clients use:

Terminal
hive --service metastore &     # remote mode only
hiveserver2 &                  # JDBC/ODBC endpoint on port 10000

HiveServer2 is the process that accepts connections from Beeline, JDBC drivers and BI tools. The next lesson connects to it.

Easier ways to get a sandbox

Building everything by hand is educational but slow. Two shortcuts:

  1. Docker: the Apache project publishes a hive image that bundles HiveServer2 and an embedded metastore, which starts in seconds.
  2. A distribution sandbox: vendors package Hive with Hadoop in a virtual machine.

With a container you only need docker run with a port mapping for 10000 and 10002 (the web UI), then connect with Beeline.

Practice with a stand-in

Hive keeps its own system tables in the metastore; SQLite has a similar built-in catalog. This stand-in lists the tables of the bookstore database from the catalog, which is the same idea as SHOW TABLES:

SQL Stand-in: listing tables from a catalog (SQLite, not Hive) ยท database: bookstore

Press Run (or Ctrl + Enter) to see the result here.

In Hive the equivalent is simply SHOW TABLES;, which reads the list from the metastore.

Common installation mistakes

  • Forgetting schematool -initSchema, then wondering why Hive will not start.
  • Using a Hive and Hadoop pair with mismatched versions (check the compatibility table in the release notes).
  • Opening two CLI sessions against the embedded Derby metastore and hitting a lock error.
  • Putting the JDBC driver in the wrong place, so Hive cannot reach MySQL.
  • Missing write permission on /tmp or the warehouse folder.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which command creates the metastore tables the first time?

    schematool initializes the metastore schema in the backing database.
  2. 2. Why is the default embedded Derby setup unsuitable for teams?

    An embedded Derby database allows only one process to open it.
  3. 3. In which mode do clients connect to a Thrift service via hive.metastore.uris?

    Remote mode uses a separate Hive Metastore Service.
  4. 4. Where does Hive keep managed table data by default?

    hive.metastore.warehouse.dir defaults to /user/hive/warehouse.

Summary

  • Hive needs Java, Hadoop (or compatible storage), a metastore database and the Hive release.
  • Create the warehouse folders and run schematool -initSchema before the first start.
  • hive-site.xml holds the key settings; embedded, local and remote describe where the metastore runs.
  • HiveServer2 (port 10000) is the endpoint for Beeline, JDBC and BI tools; a Docker image is the quickest sandbox.

Next: Hive CLI and Beeline