Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 47 ยท SQL and NoSQL on Hadoop

Pig Latin Dataflows

Learn Apache Pig's dataflow language: relations, LOAD, FILTER, GROUP, JOIN and FOREACH, user-defined functions and how Pig differs from Hive, with a Python stand-in.

Apache Pig was created at Yahoo so that people could write data-processing jobs on Hadoop without writing Java MapReduce code. Its language, Pig Latin, is not SQL. A Pig script is a dataflow: you take a dataset, transform it step by step, give each step a name, and finally store a result. Pig compiles the whole script into a plan of MapReduce (or Tez) jobs.

Pig is much less used today than Hive and Spark, but you will meet it in older pipelines, and its dataflow style survives in Spark's DataFrame API. Learning it also shows, very clearly, what a chain of transformations looks like before the engine turns it into stages.

Pig needs a Hadoop cluster (or at least its local mode), so every Pig Latin script here is read-only. The runnable examples are Python stand-ins: a small class that records transformations lazily and runs them at the end, the way Pig does. They are not Pig and do not support its full data model.

Relations, bags and tuples

Pig's data model has four pieces:

Pig data model

Atom One value, such as 42 or "Ada".
Tuple An ordered row of fields.
Bag A collection of tuples; may hold duplicates.
Map Key-value pairs inside a field.

A relation is an outer bag, basically a table.

1 / 2

Every statement assigns a new relation (a named dataset) from an existing one. Because Pig is lazy, defining relations only builds a plan. Work starts when you DUMP the result to the screen or STORE it to HDFS.

The core operators

Here is a complete script, read-only. The input is a tab-separated file of page views:

PIG
logs = LOAD '/data/pageviews' USING PigStorage('\t')
       AS (user:chararray, url:chararray, bytes:int);

big      = FILTER logs BY bytes > 1000;
by_user  = GROUP big BY user;
totals   = FOREACH by_user GENERATE
               group AS user,
               COUNT(big) AS hits,
               SUM(big.bytes) AS total_bytes;
ranked   = ORDER totals BY total_bytes DESC;
top3     = LIMIT ranked 3;

STORE top3 INTO '/out/top_users' USING PigStorage(',');

Read it top to bottom:

  • LOAD ... AS (schema) reads files and applies a schema (schema on read, as in Hive). The schema is optional; without it, fields are untyped and accessed by position ($0, $1).
  • FILTER ... BY keeps matching tuples, like WHERE.
  • GROUP ... BY collects tuples with the same key. The result has two fields: group (the key) and a bag named after the original relation containing all of its tuples. This is the MapReduce shuffle: the key is the shuffle key.
  • FOREACH ... GENERATE transforms each tuple: choose, compute or aggregate fields. After a GROUP, COUNT(big) and SUM(big.bytes) aggregate the bag.
  • ORDER, LIMIT, DISTINCT and UNION do what their names suggest.
  • JOIN combines relations on a key:
PIG
users  = LOAD '/data/users' AS (id:chararray, country:chararray);
joined = JOIN logs BY user, users BY id;          -- inner join
left   = JOIN logs BY user LEFT OUTER, users BY id;
-- after a join, fields are prefixed: logs::url, users::country
by_c   = GROUP joined BY users::country;

Useful debugging commands in the interactive shell, Grunt: DESCRIBE totals; prints a relation's schema, EXPLAIN totals; shows the logical, physical and MapReduce plans, ILLUSTRATE totals; runs the script on a tiny sample and shows each step's data. You can test a script on one machine with pig -x local script.pig; on a cluster it is pig -x mapreduce (or tez).

What the compiler does

A GROUP or JOIN or ORDER needs data grouped by key, so each becomes a shuffle (a reduce phase). Operators like FILTER and FOREACH can be pipelined into the map side or into the previous reducer. In the script above, LOAD, FILTER and the beginning of GROUP form a map phase. The aggregation FOREACH is the reduce. ORDER adds another job (it first samples to find range boundaries, so output is globally sorted), and LIMIT another. Pig's optimizer also pushes filters earlier and uses combiners for algebraic functions like SUM and COUNT (see Inside the Shuffle and Sort).

A Python stand-in

The class below mimics the style: each method returns a new relation that remembers its step, and dump() runs the pipeline. It shows laziness and what each operator does to the data.

Python A lazy Pig-style pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first print happens before (LOAD runs now): the plan exists but no data has been read. dump() is the only point of execution, exactly like DUMP and STORE in Pig. Because the plan is available before running, a real engine can reorder and combine steps. This idea returns in Spark, Flink and What Came After MapReduce.

Joins as a pipeline

A Pig JOIN is the reduce-side join from the MapReduce lesson: tag each record with its source, shuffle by the join key, and combine at the reducer. This stand-in does it explicitly.

Python JOIN as tag, shuffle and combine

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

User dee has page views but no user record, so the inner join drops her and the left outer join keeps her with None (Pig's null). User cy has no logs and appears in neither.

User-defined functions

When the built-ins are not enough, you write a UDF. Java is the native choice; Jython and other languages are also supported. Pig functions can be used in FOREACH or FILTER:

PIG
REGISTER 'udfs.py' USING jython AS myfuncs;

clean = FOREACH logs GENERATE user, myfuncs.normalize_url(url) AS url, bytes;
Python
# udfs.py (Jython UDF, read-only)
@outputSchema("url:chararray")
def normalize_url(url):
    return url.split("?")[0].rstrip("/").lower()

The @outputSchema decorator declares the result type. Remember that a UDF runs inside the map or reduce tasks, once per tuple, so keep it fast and free of external calls.

Pig versus Hive

Pig Hive
Style Procedural dataflow, one named step at a time Declarative SQL
Users Engineers building pipelines Analysts and BI tools
Schema Optional, declared in the script Stored in the metastore
Nested data First-class bags and tuples Arrays, maps, structs
Ecosystem Small and shrinking Large, shared catalog
Today Legacy ETL Common; Spark SQL is the main alternative

Pig is easier than SQL when a job is naturally a long chain of steps with many intermediate results to inspect. Hive wins whenever others need to query the data with standard tools. Most new work uses SQL engines or Spark, but the idea of a plan built lazily from named steps is the heart of all of them.

Hive, Pig and MapReduce side by side

You now know three ways to process data on Hadoop, and they sit at three different levels of abstraction. MapReduce is the raw engine: you write Java (or Streaming scripts) for mappers and reducers and control everything. Pig and Hive are higher-level layers that compile a short program into one or more MapReduce jobs (or into Tez or Spark jobs, see Hive Performance).

Aspect MapReduce Pig Hive
Level low: map and reduce functions medium: dataflow script high: declarative SQL
Language Java, or any language via Streaming Pig Latin HiveQL
Typical user developers data engineers building pipelines analysts and BI tools
Amount of code the most (a join or group-by takes dozens of lines) a few lines per step one query
Schema none (you parse bytes yourself) optional, declared in the script required, stored in the metastore
Fine control full (custom partitioners, formats, memory) moderate (UDFs, hints) limited (settings and UDFs)
Best at unusual algorithms, custom formats multi-step ETL with inspectable intermediate results ad hoc and scheduled SQL, data warehousing
Interactive use no no (batch) no by default; faster on Tez, LLAP or via Impala and Trino
Future stable, rarely written by hand legacy, shrinking large, plus Spark SQL

Hive: features and limitations

Hive's strengths are SQL familiarity, a shared metastore that other engines reuse, partitioning and bucketing, many file formats (text, ORC, Parquet, Avro, JSON through SerDes), user-defined functions, and integration with BI tools through JDBC and ODBC. Hive 3 adds ACID transactions on managed ORC tables (row-level INSERT, UPDATE, DELETE, MERGE), materialized views, and a query cache.

Its limits are just as important. It is a batch system: even on Tez a query has startup overhead, so it is not for sub-second dashboards or serving applications. Transactions work only on specific table types and carry compaction cost; it is not an OLTP database. SQL support is broad but not the whole standard, and some dialect details differ from PostgreSQL or MySQL. Performance depends heavily on table design (formats, partitions, statistics), so a badly laid out table is slow whatever the engine (see Partitioning, Bucketing and File Layout). For interactive speed, add an MPP engine (see Interactive SQL: Presto/Trino and Impala).

The same question, three ways

The task: for orders, find the total amount per customer and keep customers above 100. The Python below does it in a MapReduce style (explicit map, shuffle, reduce), in a Pig style (a chain of named relations), and in SQL (using SQLite as a stand-in for Hive).

Python One query at three levels of abstraction

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Pick the layer that matches the problem: SQL (Hive, or Spark SQL) first, because it is the shortest and the most widely understood; Pig-style dataflow only when maintaining existing scripts; hand-written MapReduce only for algorithms that SQL and dataflow cannot express.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. When does a Pig script actually process data?

    Statements only build a plan. DUMP and STORE trigger the compilation into jobs and the run.
  2. 2. What does GROUP logs BY user produce?

    GROUP collects all tuples for a key into a bag. You then aggregate it in FOREACH ... GENERATE.
  3. 3. Which Pig operators usually force a shuffle (a reduce phase)?

    They need equal keys (or a sorted range) to meet in one place. FILTER and FOREACH work row by row on the map side.
  4. 4. What is a good reason to choose Hive over Pig for a new shared dataset?

    Hive's declarative SQL and shared catalog make data accessible to many tools, which is why it displaced Pig for most uses.

Summary

  • Pig Latin describes a dataflow: named relations, each derived from earlier ones, with LOAD and STORE at the ends.
  • The core operators are FILTER, GROUP, FOREACH ... GENERATE, JOIN, ORDER, LIMIT and DISTINCT.
  • GROUP creates a bag per key; GROUP, JOIN and ORDER become shuffles, while FILTER and FOREACH pipeline.
  • Pig is lazy: nothing runs until DUMP or STORE, so the whole plan can be optimized.
  • UDFs extend Pig in Java or Jython; Hive and Spark have largely replaced Pig for new work.

Next: HBase: Wide-Column Storage.