Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 36 ยท Distribution and Ports

Distributed Erlang: nodes and clusters

Connect Erlang nodes into a cluster: node names, magic cookies, net_adm:ping, sending to remote pids, rpc calls, global names, monitor_node and security.

An Erlang node is one running instance of the virtual machine. Because processes communicate only by messages, the language makes a remarkable promise: a message sent to a process on another machine uses exactly the same ! operator. That location transparency is built in, not added by a library. A group of connected nodes is a cluster.

Naming a node

Start the VM with a name to make it distributed:

TEXT
$ erl -sname alpha            # short name: alpha@hostname, same network
$ erl -name [email protected]    # long name: fully qualified or IP

The shell prompt then shows the node name: (alpha@laptop)1>. In code, node() returns the node's own name and nodes() lists the nodes it is connected to. Short names and long names cannot be mixed in one cluster.

Nodes connect when one uses the other. The simplest way:

ERLANG
(alpha@laptop)1> net_adm:ping(beta@laptop).
pong
(alpha@laptop)2> nodes().
[beta@laptop]

pong means connected; pang means the node was not reachable. Connections are transitive by default: if alpha is connected to beta, and beta to gamma, alpha and gamma will also connect, which forms a fully meshed cluster. (That limits clusters to roughly dozens to a few hundred nodes; see -connect_all false and the hidden option for large setups.)

Access is guarded by a magic cookie, a shared secret atom. Two nodes may connect only if they hold the same cookie. By default it is read from the file ~/.erlang.cookie, or you can set it with -setcookie secret or erlang:set_cookie/2.

The cookie is authentication, not encryption, and a node with the cookie can run any code on every other node. Never expose the distribution port (EPMD on 4369 plus the dynamic port range) to the internet. Use a private network, firewall rules and, for untrusted networks, TLS distribution (-proto_dist inet_tls).

Sending to remote processes

Pids work across nodes, and a registered name is addressed as {Name, Node}:

ERLANG
%% on alpha
(alpha@laptop)3> {logger, beta@laptop} ! {log, "hello from alpha"}.
{log,"hello from alpha"}
(alpha@laptop)4> Pid = spawn(beta@laptop, io, format, ["running on ~p~n", [node()]]).
running on beta@laptop
<8372.90.0>

spawn/4 starts a process on another node (the module must exist there). The printed pid <8372.90.0> has a non-zero first number, which marks it as remote. Linking and monitoring remote processes work as before; if the connection breaks, links deliver noconnection exit reasons and monitors a DOWN with the same reason.

You can also monitor a whole node:

ERLANG
monitor_node(beta@laptop, true),
receive {nodedown, beta@laptop} -> alert() end.

or subscribe to all changes with net_kernel:monitor_nodes(true).

Remote procedure calls

erpc (modern) and rpc run a function on another node and return the result:

ERLANG
erpc:call(beta@laptop, erlang, system_time, [second]).
%% 1767225600     (example value)
erpc:call(beta@laptop, mymod, compute, [10], 5000).   % 5 s timeout
rpc:multicall([beta@laptop, gamma@laptop], erlang, memory, [total]).

erpc:call raises an exception on failure (including {erpc, noconnection} and {erpc, timeout}); the older rpc:call returns {badrpc, Reason} instead.

Global registration and process groups

The local name registry is per node. To find a service anywhere in the cluster:

  • global:register_name(Name, Pid) and global:whereis_name(Name): a cluster-wide unique name, with conflict resolution when partitions heal.
  • pg (process groups): any number of processes join a named group on any node, and you can fetch all members to broadcast to them.
ERLANG
pg:start_link(),
pg:join(chat_room, self()),
[Member ! {say, "hi"} || Member <- pg:get_members(chat_room)].

Building a cluster

Give names erl -sname alpha
Share a cookie Same secret on every node
Connect net_adm:ping(Node)
Send as usual {Name, Node} ! Msg

From here, ! works across machines.

1 / 2

Failure in a cluster

Networks fail in ways a single machine does not. Messages are delivered at most once, with no acknowledgement. A send to an unreachable node does not raise; the message vanishes. Design for that:

  • Use monitors with timeouts and idempotent operations that are safe to retry.
  • Expect partitions: two halves of a cluster each believe the other is down. Pick a policy (the larger half wins, a quorum, or last-writer-wins) before it happens. Libraries such as ra (Raft) exist for consistent state.
  • Heartbeats between nodes are configured via net_ticktime (default 60 s), which controls how fast a dead node is detected.

Run it: a two-node stand-in

Distribution cannot run here, so this is a JavaScript stand-in, not Erlang. Each "node" has its own process table; a "connection" is only allowed with a matching cookie; and sending to a disconnected node silently loses the message, as in real Erlang.

JS Stand-in: cookies and remote sends

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Common mistakes

  • Mixing -sname and -name nodes: they cannot connect.
  • Different cookies on different machines, or a world-readable cookie file (~/.erlang.cookie must be mode 400).
  • Treating remote calls like local ones: always set timeouts and handle noconnection.
  • Assuming that a message was received because send returned.
  • Opening distribution ports to untrusted networks.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the purpose of the Erlang magic cookie?

    The cookie is a shared secret for authentication; it does not encrypt traffic.
  2. 2. How do you send a message to a process registered as logger on node beta@host?

    A remote registered process is addressed as the tuple {Name, Node}.
  3. 3. What does net_adm:ping/1 return for an unreachable node?

    pong means connected; pang means not reachable.
  4. 4. What happens when you send a message to a node that just went down?

    Sends are asynchronous and at-most-once; use monitors and timeouts to detect failures.

Summary

  • A node is a named VM; connecting nodes forms a cluster, secured only by a shared cookie.
  • ! works unchanged across nodes: use pids or {Name, Node}.
  • erpc:call runs functions remotely; global and pg locate services cluster-wide.
  • Networks fail: use monitors, timeouts and idempotent operations, and plan for partitions.
  • Protect distribution ports; use a private network or TLS.

Next: Ports and sockets