Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 36 ยท UDFs and Transactions

Writing Java UDFs, UDAFs and UDTFs

Build and register Hive user-defined functions in Java: a GenericUDF, the UDAF aggregation lifecycle, a UDTF, packaging a JAR, CREATE FUNCTION and testing without a cluster.

When the built-in functions cannot express your logic, you write a user-defined function in Java, compile it into a JAR, and register it with Hive. This lesson walks through the three kinds, the packaging, and a way to test the core logic without a cluster.

The Hive classes (org.apache.hadoop.hive...) are not available on this site, so the Hive code is read-only. The runnable Java editors contain only the plain-Java logic that the UDF would call, which is exactly how you should unit-test real UDFs: keep the logic in a method that does not depend on Hive types.

Project setup

A UDF project is an ordinary Maven or Gradle project with Hive as a provided dependency, so the JAR does not bundle Hive itself:

XML
<dependency>
  <groupId>org.apache.hive</groupId>
  <artifactId>hive-exec</artifactId>
  <version>4.0.1</version>   <!-- match your cluster -->
  <scope>provided</scope>
</dependency>

Match the version and Java level of your cluster. Build with mvn package, and you get target/my-udfs-1.0.jar.

A scalar function with GenericUDF

Hive has a legacy UDF base class with a simple evaluate method. It is deprecated, so write new functions as a GenericUDF. It is a little more code, but it handles type checking, NULL, constant arguments and complex types properly.

Java
package com.example.hive;

import org.apache.hadoop.hive.ql.exec.Description;
import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
import org.apache.hadoop.hive.ql.exec.UDFArgumentLengthException;
import org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException;
import org.apache.hadoop.hive.ql.metadata.HiveException;
import org.apache.hadoop.hive.ql.udf.generic.GenericUDF;
import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.StringObjectInspector;

@Description(name = "mask_email",
             value = "_FUNC_(str) - hides the local part of an email address",
             extended = "Example: SELECT mask_email('[email protected]') -> 'a**@example.com'")
public class MaskEmail extends GenericUDF {

  private StringObjectInspector input;

  @Override
  public ObjectInspector initialize(ObjectInspector[] args) throws UDFArgumentException {
    if (args.length != 1) {
      throw new UDFArgumentLengthException("mask_email takes exactly 1 argument");
    }
    if (!(args[0] instanceof StringObjectInspector)) {
      throw new UDFArgumentTypeException(0, "mask_email expects a string");
    }
    input = (StringObjectInspector) args[0];
    return PrimitiveObjectInspectorFactory.javaStringObjectInspector;   // return type
  }

  @Override
  public Object evaluate(DeferredObject[] args) throws HiveException {
    Object raw = args[0].get();
    if (raw == null) {
      return null;
    }
    String s = input.getPrimitiveJavaObject(raw);
    int at = s.indexOf('@');
    if (at < 2) {
      return s;
    }
    return s.charAt(0) + "*".repeat(at - 1) + s.substring(at);
  }

  @Override
  public String getDisplayString(String[] children) {
    return "mask_email(" + children[0] + ")";
  }
}

The two phases matter:

  • initialize runs once at query compile time. It checks argument types and declares the return type through an ObjectInspector, Hive's description of a type.
  • evaluate runs once per row. Keep it fast, avoid allocating heavily, and always handle NULL.

The core of evaluate is just string handling. Because it does not touch Hive types, you can test it as a plain Java method:

Java Core logic of mask_email (plain Java stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

This is a stand-in, but it tests the edge cases (NULL, too short, no @) that cause most production UDF bugs.

Registering a function

SQL
-- Session-only
ADD JAR hdfs:///udfs/my-udfs-1.0.jar;
CREATE TEMPORARY FUNCTION mask_email AS 'com.example.hive.MaskEmail';

-- Permanent, stored in the metastore, tied to a database
CREATE FUNCTION marketing.mask_email
  AS 'com.example.hive.MaskEmail'
  USING JAR 'hdfs:///udfs/my-udfs-1.0.jar';

SELECT mask_email(email) FROM customers;
SELECT marketing.mask_email(email) FROM customers;

DROP FUNCTION marketing.mask_email;
RELOAD FUNCTION;   -- pick up functions created by another HiveServer2 instance

Keep the JAR on HDFS or an object store so every node can fetch it. When you deploy a new version, use a new file name or a new function name; clusters cache loaded classes and a replaced JAR may not be picked up until the session restarts.

Aggregation with a UDAF

An aggregate runs in parts: partial aggregates are built on mappers, merged, and finalized on reducers. You write an AbstractGenericUDAFResolver that returns a GenericUDAFEvaluator, and the evaluator implements the lifecycle:

Method Called Purpose
init(mode, inputs) once per mode declare input and output types
getNewAggregationBuffer() per group create an empty state object
iterate(buffer, params) per input row fold one row into the buffer
terminatePartial(buffer) end of the map side emit the partial result
merge(buffer, partial) reduce side combine partial results
terminate(buffer) end produce the final value

The mode (PARTIAL1, PARTIAL2, FINAL, COMPLETE) tells init which of those steps this instance will perform. The key design rule is that the partial result must be mergeable: you cannot compute a median from partial medians, but you can compute an average from partial (sum, count) pairs. This plain-Java stand-in does a geometric mean using the same lifecycle names, with three "mappers" and one "reducer":

Java UDAF lifecycle (plain Java stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Both numbers agree, which proves the partial result is mergeable. If they disagreed, your UDAF would give different answers depending on how data is split, a classic bug.

Table-generating functions (UDTF)

A UDTF extends GenericUDTF and implements initialize (declare the output columns as a struct), process (called per input row; call forward(Object[]) once per output row) and close.

Java
public class SplitWords extends GenericUDTF {
  @Override
  public StructObjectInspector initialize(StructObjectInspector argOIs) throws UDFArgumentException {
    List<String> names = Arrays.asList("word", "pos");
    List<ObjectInspector> types = Arrays.asList(
        PrimitiveObjectInspectorFactory.javaStringObjectInspector,
        PrimitiveObjectInspectorFactory.javaIntObjectInspector);
    return ObjectInspectorFactory.getStandardStructObjectInspector(names, types);
  }

  @Override
  public void process(Object[] args) throws HiveException {
    if (args[0] == null) return;
    String[] words = args[0].toString().split("\\s+");
    for (int i = 0; i < words.length; i++) {
      forward(new Object[] { words[i], i });
    }
  }

  @Override public void close() { }
}

Use it through LATERAL VIEW, as with explode in the previous lesson:

SQL
SELECT r.id, w.word, w.pos
FROM reviews r
LATERAL VIEW split_words(r.text) w AS word, pos;

Performance and safety tips

  • Prefer built-ins and CASE expressions to custom code whenever they can do the job.
  • Be deterministic. Annotate with @UDFType(deterministic = true) (the default) so Hive can cache and push down. Mark a function with deterministic = false if it uses randomness or the clock, or constant folding will break it.
  • Avoid state across rows in plain UDFs; Hive may run them in any order and in parallel.
  • Test the logic as plain Java (as above) and the Hive wrapper with hive-exec test utilities or a small local Hive test.
  • Security. A UDF is arbitrary code running in your cluster. Allow only reviewed JARs, and restrict ADD JAR and CREATE FUNCTION with the authorization settings described in security and HiveServer2.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. When does GenericUDF.initialize run?

    initialize validates inputs and returns an ObjectInspector for the result type; evaluate does the per-row work.
  2. 2. Which UDAF method combines partial results from different tasks?

    merge folds a partial aggregate produced by another task into the current buffer.
  3. 3. Why is a median hard to write as a simple UDAF?

    A partial result must be mergeable without losing information; sum and count merge, medians do not.
  4. 4. What is the difference between CREATE TEMPORARY FUNCTION and CREATE FUNCTION?

    A temporary function disappears when the session ends; a permanent one is recorded in the metastore and visible to everyone allowed to use it.

Summary

  • Write new scalar functions as GenericUDF: initialize at compile time, evaluate per row, always handle NULL.
  • UDAFs follow the iterate, terminatePartial, merge, terminate lifecycle; partial results must be mergeable.
  • UDTFs call forward once per output row and are used with LATERAL VIEW.
  • Register with ADD JAR and CREATE [TEMPORARY] FUNCTION ... AS 'class' [USING JAR '...'].
  • Keep core logic in plain Java so you can test it without a cluster.

Next: ACID Transactions in Hive