Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 36 ยท Tools and Testing

Debugging, tracing and the Observer

Inspect running Erlang systems: io:format and logger, process_info, the Observer GUI, dbg tracing, the recon library, crash dumps and finding a mailbox leak.

Erlang was built for systems that never stop, so its tools are built for inspecting a live system. You can look inside a running node, trace a function call as it happens and find the process that is eating memory, all without restarting. This lesson walks from the simplest tools to the powerful ones.

The first tool is the oldest: io:format("value: ~p~n", [Value]). ~p pretty-prints any term, ~w prints it plainly, ~s prints a string, ~n is a newline. It writes to the group leader of the process, which in a release may be a remote shell.

For anything that stays in the code, use logger, built into OTP:

ERLANG
-include_lib("kernel/include/logger.hrl").

handle(Req) ->
    ?LOG_INFO("request ~p", [Req]),
    ?LOG_ERROR(#{what => bad_request, request => Req}).

Levels (debug, info, notice, warning, error, critical, alert, emergency) can be changed at run time: logger:set_primary_config(level, debug). Crashes of OTP processes automatically produce crash reports with the process state and a stack trace, a gold mine; make sure the sasl application or default logger handlers are active.

Reading a stack trace

ERLANG
1> lists:nth(5, [a, b]).
** exception error: no function clause matching lists:nth(5,[a,b])
2> erlang:get_stacktrace().   % removed; use catch Class:Reason:Stacktrace

Use the modern syntax to capture it:

ERLANG
try risky()
catch Class:Reason:Stack ->
    ?LOG_ERROR("~p:~p~n~p", [Class, Reason, Stack])
end.

Class is error, exit or throw. The stack lists {Module, Function, Arity, [{file, F}, {line, L}]}, newest call first.

Looking at processes

ERLANG
process_info(Pid).                                 % everything
process_info(Pid, [message_queue_len, memory, current_function,
                   registered_name, status, links, monitors]).
erlang:process_info(Pid, current_stacktrace).
sys:get_state(Pid).                                % state of a gen_server
sys:trace(Pid, true).                              % print each message in/out

sys works on any OTP behaviour process, and sys:get_status(Pid) includes its callback module and state. In the shell, i(). lists all processes and regs(). the registered ones.

Common symptoms and where to look:

  • A process with a huge message_queue_len: it cannot keep up, or it is stuck in a receive that never matches.
  • A big memory: a large state, leaked binaries or a long list.
  • status waiting for a long time while current_function shows gen:do_call: it is blocked in a call to another server, which is how deadlocks look.

Observer

observer:start() opens a graphical tool (needs wx support) that shows system load, memory, the application tree, a sortable process list, ETS tables and tracing. It can also attach to a remote node (menu Nodes) if you know its name and cookie, which is great for production diagnosis. On a headless server use observer_cli or recon from a remote shell instead. The Application tab draws the supervision tree you built, nice for checking that processes sit where you expect.

A debugging ladder

io:format, logger See values and events
process_info, sys Inspect a running process
observer, recon Find the heavy process
dbg, trace Watch calls and messages live

Start at the top; go down only when needed.

1 / 2

Tracing with dbg and trace

Tracing shows function calls, messages and process events without changing code. The older dbg module is easy to use in a shell:

ERLANG
1> dbg:tracer().
2> dbg:p(all, c).                              % trace calls in all processes
3> dbg:tpl(lists, reverse, x).                 % trace lists:reverse, including return and exceptions
4> lists:reverse([1, 2, 3]).
(<0.80.0>) call lists:reverse([1,2,3])
(<0.80.0>) returned from lists:reverse/1 -> [3,2,1]
5> dbg:stop().

(Output format is illustrative; pids differ.) Always stop tracing and be careful: tracing everything on a busy production node floods it. Production tools wrap dbg with limits. The recon library offers safe helpers:

ERLANG
recon:proc_count(memory, 5).                   % top 5 processes by memory
recon:proc_count(message_queue_len, 5).        % top 5 by mailbox size
recon:bin_leak(5).                             % who holds on to binaries
recon_trace:calls({kv, put, '_'}, 10).         % trace 10 calls, then stop

Limits make it safe: the trace stops by itself after the number of events you give.

The debugger and crash dumps

debugger:start() opens a GUI debugger that supports breakpoints, but needs modules compiled with debug_info and is rarely used on concurrent code. In practice, tracing and logging beat breakpoints, which freeze a process while its partners time out.

When the whole VM dies, it can write erl_crash.dump. Open it with crashdump_viewer:start() to see the reason (for example "eheap_alloc: Cannot allocate" for memory exhaustion), process list and message queues. The ERL_CRASH_DUMP environment variable sets its location.

Run it: finding the process with a runaway mailbox (stand-in)

This JavaScript stand-in is not Erlang; it mimics recon:proc_count(message_queue_len, N). You can build a table of processes and sort by mailbox length to find the one that cannot keep up.

JS Stand-in: top processes by mailbox

Press Run (or Ctrl + Enter) to see the console output here.

No output.

A call tracer stand-in

Tracing without touching the function body: wrap it, record the call and return value, and stop after a fixed number of events, like recon_trace.

JS Stand-in: limited call tracing

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Habits that save hours

  • Reproduce first, then trace; do not guess.
  • Check message_queue_len before blaming the CPU.
  • Name processes and give them readable ids; registered_name in tools beats raw pids.
  • Log structured maps, not formatted text, so tools can filter.
  • Keep debug_info on in releases so tools can show source lines.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does ~p do in io:format?

    ~p pretty-prints a term; ~w prints plainly and ~s prints a string.
  2. 2. Which call returns the internal state of a running gen_server?

    The sys module works with all OTP behaviours.
  3. 3. A process shows a huge message_queue_len. What is the most likely meaning?

    Messages accumulate when processing is slow or patterns do not match.
  4. 4. Why does recon_trace take a call limit?

    Unbounded tracing can flood and overload a live system.

Summary

  • Use io:format for quick looks and logger for lasting logs and crash reports.
  • process_info, sys:get_state and sys:trace inspect live processes; watch mailbox length and memory.
  • observer gives a GUI over load, processes and supervision trees, locally or on a remote node.
  • dbg and recon_trace trace calls without changing code; always limit and stop traces.
  • Crash dumps and crashdump_viewer explain a dead VM.

Next: Types, specs and Dialyzer