Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 40 ยท Metaprogramming

Macros and Metaprogramming

Learn how Julia code is data: build and inspect Expr trees, use quote and interpolation, write hygienic macros with esc, and see how @time and @assert work.

Julia is homoiconic in spirit: a piece of code is represented by an ordinary Julia value that a program can inspect, build and change. Writing programs that manipulate programs is called metaprogramming. It sounds exotic, but you already use it. @time, @assert, @show and @btime are macros, and so are the special string literals like r"\d+".

In this lesson you will meet the Expr type, learn how quote and $ build code, write your own macro and understand hygiene, and see what @macroexpand shows. Julia cannot run here, so its code is read-only. The Python stand-ins use Python's own ast module, which also treats code as a tree, to show the same ideas: parsing, inspecting, transforming and evaluating.

Code is data: Expr

When Julia parses 1 + 2 * 3, it builds an expression tree. You can capture it without running it by quoting with :( ... ):

JULIA
ex = :(1 + 2 * 3)

typeof(ex)         # Expr
ex.head            # :call
ex.args            # Any[:+, 1, :(2 * 3)]
eval(ex)           # 7

An Expr has a head that says what kind of node it is (:call, :if, :block, :(=)) and a vector args with its children. Children are other Expr values, symbols such as :x (names), or literal values such as 1. Operators are just function calls in the tree: 1 + 2 is the call +(1, 2).

JULIA
dump(:(x + 1))
# Expr
#   head: Symbol call
#   args: Array{Any}((3,))
#     1: Symbol +
#     2: Symbol x
#     3: Int64 1

You can build the same tree by hand, and since it is data you can edit it:

JULIA
ex = Expr(:call, :+, 1, :(2 * 3))     # same as :(1 + 2 * 3)
ex.args[1] = :*                        # change + into *
eval(ex)                               # 1 * (2 * 3) = 6

Meta.parse("1 + 2") turns a string into an Expr, and eval runs an expression in the current module. Use eval sparingly: it runs at global scope and is slow compared with a normal function call.

Code as data

Source text 1 + 2 * 3
Expr tree A head and child arguments.
Transform Rewrite the tree like any data.
eval Run the new code.
1 / 2

Code is data in Python, too

Python exposes the same idea through the ast module. It parses source into a tree of node objects that you can walk, change and compile. The stand-in below mirrors the Julia examples: parse, look at the tree, change the operator and run the result.

Python Parse, inspect, change and run code

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

quote, interpolation and the macro

Writing Expr(:call, ...) by hand is tedious, so Julia gives you quoting. :(...) quotes one expression, and quote ... end quotes a block of several statements. Inside a quote, $ interpolates a value or expression into the tree:

JULIA
x = 5
:(y = $x + 1)               # :(y = 5 + 1), the value of x is spliced in
:(y = x + 1)                # :(y = x + 1), no interpolation

op = :*
:($op(2, 3))                # :(2 * 3), a symbol becomes the function name

A macro is a function that runs at parse time. Its inputs are the expressions that appear after the macro name, not their values, and its result is a new expression that replaces the macro call in the program.

JULIA
macro twice(ex)
    return quote
        $(esc(ex))
        $(esc(ex))
    end
end

n = 0
@twice n += 1
n                  # 2

The call @twice n += 1 is replaced, before compiling, by two copies of n += 1. The esc part is explained in a moment. You call macros without parentheses by default (@twice n += 1), or with them (@twice(n += 1)), and @macroexpand shows what they turn into:

JULIA
@macroexpand @twice n += 1
# quote
#     n += 1
#     n += 1
# end

How the built-in macros work

Two macros you use daily are macros for good reason:

  • @assert cond msg must print the text of the condition when it fails. A function sees only the value false, whereas a macro sees the expression x > 0 and can turn it into a message. @assert x > 0 fails with AssertionError: x > 0.
  • @time expr must start a timer before the expression runs, so it needs to wrap the unevaluated code. A function would receive an already-computed result.

Here is a miniature version of @assert to see the mechanism:

JULIA
macro check(cond)
    text = string(cond)                          # the source text of the condition
    return :( $(esc(cond)) || error("check failed: ", $text) )
end

x = -3
@check x > 0      # ERROR: check failed: x > 0

The same trick in Python works on the tree. This stand-in "expands" a check(...) call into an if not ...: raise statement, using the original source text for the message:

Python A tiny assert macro, written on the tree

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In both languages the key point is the same: the transformation happens on code, before it runs, so it can use the source text and wrap evaluation. Python has no macros in the language, so we ran the transformation by hand.

Hygiene and esc

A good macro must not accidentally capture or overwrite the caller's variables. Julia macros are hygienic: any variable that the macro itself introduces is automatically renamed to a unique name, so it cannot clash. Names that come from the user's code must be left alone, and you ask for that with esc.

JULIA
macro timeit(ex)
    return quote
        local t0 = time_ns()                 # hygienic: renamed, cannot clash with the caller's t0
        local result = $(esc(ex))            # the user's code: resolved in the caller's scope
        local elapsed = (time_ns() - t0) / 1e9
        println("took ", elapsed, " s")
        result
    end
end

t0 = "my own variable"
@timeit sum(1:1_000_000)
t0                                           # still "my own variable"

Without esc, a name inside ex such as data would be looked up in the module where the macro was defined, not where it was used, and the user's local variables would be invisible. The rule of thumb:

  • Variables and expressions that belong to the user, escape with $(esc(x)).
  • Temporaries that belong to the macro, leave unescaped (or use gensym()).
  • Global functions the macro calls, such as println, resolve in the macro's own module, which is what you want.

You can print a macro's output with @macroexpand to see the renamed variables, which appear as names like var"#12#t0".

Other metaprogramming tools

  • @eval evaluates an expression in a module, and is handy for generating many similar definitions in a loop:
JULIA
for name in (:red, :green, :blue)
    @eval $(Symbol(name, "_text"))(s) = string($(string(name)), ": ", s)
end
green_text("go")        # "green: go"
  • String macros such as r"[a-z]+" (a regular expression) and raw"C:\dir" are macros named @r_str and @raw_str. Packages define their own, for instance md"# Title" for Markdown.
  • Generated functions (@generated) receive the types of their arguments and return the expression to compile for those types. They are rare and powerful: the body runs at compile time for each combination of argument types.
JULIA
@generated function tuple_length(t::Tuple)
    n = length(t.parameters)          # t is a type here, not a value
    return :($n)                      # the compiled method just returns a constant
end

tuple_length((1, "a", 2.0))           # 3
  • Function-like helpers that need no macro: @which f(1) finds the method called, Meta.@lower and @code_lowered show the lowered form, and Base.@kwdef (a macro!) generates keyword constructors for a struct, as seen in Structs and Custom Types.

When not to use a macro

Macros are harder to read, to debug and to compose than functions. Reach for one only when you need to see the unevaluated code (the text of an expression, delaying or repeating evaluation, new syntax), or to generate boilerplate that a function cannot. If a function can do the job, write a function. Many "metaprogramming" needs are better served by multiple dispatch, as covered in Multiple Dispatch.

Never eval strings from untrusted users: Meta.parse plus eval runs arbitrary Julia code with all of your program's permissions.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the type of the value :(1 + 2)?

    Quoting captures code as an Expr tree, with head :call and arguments +, 1 and 2, instead of running it.
  2. 2. What does a macro receive as input?

    Macros run at parse time on the code itself, and return new code that replaces the macro call.
  3. 3. Why does @assert x > 0 have to be a macro rather than a function?

    A function receives only the Boolean result. A macro sees the expression, so it can show it in the error message.
  4. 4. What is esc used for in a macro?

    esc(ex) marks the user's code so that its variables are resolved in the caller's scope instead of being renamed or looked up in the macro's module.
  5. 5. Which tool shows what a macro call expands into?

    @macroexpand @mymacro args prints the code that replaces the call.

Summary

  • Julia code is represented by Expr trees (a head and args) that programs can inspect, build, edit and eval.
  • quote or :( ... ) captures code, and $ interpolates values or expressions into it.
  • A macro runs at parse time on unevaluated expressions and returns new code; @macroexpand reveals the result.
  • Hygiene renames the macro's own variables; esc keeps the caller's code in the caller's scope.
  • @eval, string macros and @generated functions are further tools; prefer ordinary functions and dispatch when they suffice.

Next, handle things going wrong in Errors and Exceptions.