Skip to content
Tutorials Center
My learning

Topics

Lesson 1 of 36 ยท Getting Started

What Is Apache Hive?

Learn what Apache Hive is, why it was created, how it turns SQL-like HiveQL into distributed jobs, and how it differs from a traditional relational database.

Apache Hive is a data warehouse system built on top of a distributed storage layer such as HDFS or an object store like Amazon S3. It lets you query huge files with a SQL-like language called HiveQL, instead of writing low-level programs for a cluster. You describe what you want ("total sales per country"), and Hive works out the distributed jobs that produce the answer.

Hive was created at Facebook around 2007, when analysts needed to query terabytes of logs and the only tool was hand-written MapReduce code in Java. Writing a job for every question was slow and needed programmers. Hive gave analysts a language they already knew, and it became an Apache project in 2008.

About the examples in this course. A real Hive server (a Hadoop cluster, a metastore and a query engine) cannot run inside a web page. So every Hive-specific statement here is shown in a read-only code block. The runnable editors use portable SQL on a small bookstore database, and each one is labelled as a stand-in: it shows the same idea with SQLite, not with Hive itself. Output is printed only when we are sure of it.

The problem Hive solves

Imagine a web company that stores one billion log lines per day as plain files. A relational database would struggle to load that much data, and a single server cannot read it fast enough. Distributed storage solves the capacity problem: files are cut into blocks and spread over many machines. But now the data lives in files, not tables, and nobody can write SELECT against a folder.

Hive adds the missing layer:

  • A schema on top of files. You declare that a folder of text files is a table with named, typed columns.
  • A query language. HiveQL looks like SQL, so analysts, BI tools and applications can use it.
  • A compiler. Each query is translated into a plan that runs in parallel on the cluster.
  • A catalog. The metastore remembers which tables exist, where their files are and what the columns are.

Notice what Hive is not: it does not store your data itself, and it is not a transactional database. Your data stays in files; Hive stores only metadata about them.

A first look at HiveQL

HiveQL is close enough to SQL that you can read it immediately. This statement creates a table over delimited text files and then queries it:

SQL
CREATE TABLE page_views (
  user_id   BIGINT,
  url       STRING,
  viewed_at TIMESTAMP
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
STORED AS TEXTFILE;

SELECT url, COUNT(*) AS views
FROM page_views
GROUP BY url
ORDER BY views DESC
LIMIT 10;

The CREATE TABLE line is almost standard SQL. The ROW FORMAT and STORED AS clauses are Hive's way of telling the engine how to read the bytes in the files. You will study them in the chapter on file formats.

Because the SELECT part is plain SQL, we can practise it right now. The next editor is a stand-in: it runs the same shape of query (a group, a count and a top-N) on a bookstore database in SQLite.

SQL Stand-in: top-N per group (SQLite, not Hive) ยท database: bookstore

Press Run (or Ctrl + Enter) to see the result here.

Hive versus a traditional database

Hive looks like a database but is designed for a different job. The table summarizes the difference.

Aspect Traditional RDBMS (MySQL, PostgreSQL) Apache Hive
Main workload Many small reads and writes (OLTP) Large scans and aggregations (OLAP, batch analytics)
Data size Gigabytes to a few terabytes Terabytes to petabytes
Storage The database's own files Files in HDFS or object storage
Schema Checked when data is written Applied when data is read ("schema on read")
Latency Milliseconds Seconds to minutes per query
Updates Row-level, fast Possible with ACID tables, but not its strength
Scaling Mostly vertical Horizontal: add machines

Schema on read deserves a closer look. In a normal database, an INSERT with a bad value is rejected. In Hive you can drop any file into a table's folder; the table definition is only applied when you query, and values that do not fit the declared type come back as NULL. That makes loading very fast and flexible, but it moves the checking job to you.

Where Hive fits today

Hive grew up with MapReduce, but it is no longer tied to it. Modern Hive runs on engines such as Apache Tez (and historically Spark), which are much faster for interactive-sized queries. Even when companies move to newer engines, the Hive metastore has become the standard catalog: Spark, Trino, Presto, Flink and Impala all read table definitions from it. Learning Hive therefore teaches you concepts (partitions, file formats, external tables) that you will use across the whole data ecosystem.

flowchart LR
    A["Analyst / BI tool"] --> B["HiveQL query"]
    B --> C["Hive compiler"]
    C --> D["Distributed engine<br/>(Tez, Spark, MapReduce)"]
    D --> E["Files in HDFS / S3"]
    C --> F["Metastore<br/>(table definitions)"]

Hive turns a query into jobs, and uses the metastore to find the files for each table.

When to use Hive and when not to

Use Hive for scheduled reports, ETL pipelines, log analysis, and historical analytics over very large datasets where a few seconds of start-up cost do not matter. Avoid it for a shopping-cart database, for low-latency lookups by key, or for workloads with constant single-row updates. Use a transactional database for those.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does Hive store itself?

    Hive keeps table definitions in the metastore. The data stays in ordinary files on the storage layer.
  2. 2. What does 'schema on read' mean?

    Hive lets you add files first and interprets them with the table schema at query time.
  3. 3. Which workload is Hive best suited to?

    Hive is built for large scans and aggregations, where start-up latency is acceptable.
  4. 4. Why is the Hive metastore still important today?

    The metastore became the de facto catalog that many engines share.

Summary

  • Hive is a data warehouse layer that gives files in HDFS or object storage a table schema and a SQL-like language (HiveQL).
  • It compiles queries into distributed jobs; it is built for batch analytics, not for low-latency transactions.
  • Data stays in files; Hive keeps only metadata in the metastore, and applies the schema when data is read.
  • The examples in this course use read-only Hive code plus runnable SQLite stand-ins for portable SQL.

Next: Installing Hive and Run Modes