Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 36 ยท Communication

Messages, RPC and Delivery Guarantees

Learn how nodes talk: request/response vs messages, RPC, timeouts, at-most-once, at-least-once and the exactly-once myth, with a lossy-network simulation.

Nodes in a distributed system have no shared memory. The only way one machine can learn anything from another is by sending bytes across a network, and the network is allowed to lose, delay, duplicate and reorder them. Everything in this chapter, from retries to queues, is about living with that fact.

In this lesson you will meet the two basic communication styles (request/response and one-way messages), see why a timeout is a guess and not a fact, and learn the three delivery guarantees: at-most-once, at-least-once and "exactly-once". You will measure all three on a simulated lossy network that you can run right on the page.

Two styles: request/response and messages

Request/response is the style you know from HTTP. The caller sends a request and waits for a reply. A remote procedure call (RPC) wraps this in a function-call shape: balance = bank.get_balance(42) looks like a local call, but a network sits in the middle.

Messaging is one-way. A sender hands a message to the network (or to a broker, see queues and pub/sub) and moves on. Nobody has to be listening at that moment, and there may be many receivers or none.

Two ways to talk

Request/response Ask and wait for the answer.
Messages Send and move on.

RPC is request/response dressed up as a function call.

1 / 2

The big difference from a local call is the last card. When a local function returns, you know it ran. When a remote call gives you no answer, you know almost nothing.

The timeout is a guess

A client sends a request and starts a timer. If no reply arrives, three different worlds are possible, and from the client's side they look exactly the same:

sequenceDiagram
    participant C as Client
    participant S as Server
    Note over C,S: World 1: request lost
    C-xS: request
    Note over C,S: World 2: server slow or crashed
    C->>S: request
    Note over C,S: World 3: reply lost
    C->>S: request
    S--xC: reply

Three different failures, one identical symptom: the client's timer runs out.

In world 1 the work never happened. In world 2 it may or may not have happened. In world 3 it definitely happened. A timeout only tells you that you stopped waiting. This single ambiguity is why delivery guarantees exist.

Simulating a lossy network

Let's build the smallest possible world: a server that adds 1 to a counter, a network that drops each packet with probability 30%, and a seeded random generator so every run prints the same numbers. A drop can hit the request or the reply.

Python One call over a lossy network

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the lines that say "reply lost": the client sees a failure, yet the server's counter moved. Now we can compare strategies.

At-most-once

With at-most-once delivery the client sends the request once and never retries. If anything goes wrong the operation simply doesn't happen, or happens without the client learning about it. Nothing is ever executed twice.

This is the right choice when a duplicate would be worse than a loss: a status ping, a metrics sample, a "like" you can afford to miss. The cost is that messages can vanish.

At-least-once

With at-least-once delivery the client keeps retrying until it gets an acknowledgment. Nothing is lost (as long as the network eventually works), but a retry after a lost reply makes the server run the handler a second time. Duplicates are the price.

This is what most real systems give you by default: HTTP clients with retries, message brokers that redeliver unacknowledged messages, and job queues.

Python At-most-once vs at-least-once

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The at-most-once run executed only 673 of the 1000 increments, because requests were lost and never retried. The at-least-once run executed 1442: every time a reply was lost, the work was done again, giving 442 duplicates. Look at "confirmed" too: with one try the client heard back about only 465 operations, although the server applied 673. The client's belief about the world and the real state drift apart.

The exactly-once myth

Everybody wants exactly-once delivery: every operation happens once, no more and no less. Over an unreliable network, no protocol can guarantee that a message is delivered exactly once. The sender can never be sure the receiver got it without an acknowledgment, and the acknowledgment can itself be lost, which is the same problem again.

What systems really offer is exactly-once effect (also called effectively-once): at-least-once delivery plus a receiver that recognises duplicates and ignores them. The trick is to give each operation a unique ID, and make the server remember which IDs it has already handled.

Python Exactly-once effect = retries + deduplication

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

There were 2,123 network attempts for 1,000 operations, but the handler ran exactly once per operation, and the total is exactly 1000. Messages were delivered at least once; the effect happened exactly once. The next lesson, Idempotency, Retries and Backoff, digs into this pattern.

The dedup table has to live as long as a retry can arrive, and it must be updated in the same atomic step as the work itself. If the server crashes between "did the work" and "remembered the ID", you lose exactly-once again.

Choosing a guarantee

Guarantee Sender behavior Failure result Good for
At-most-once Send once, no retry Message may be lost Pings, metrics, best-effort notices
At-least-once Retry until acknowledged Duplicates possible Most work: jobs, events, payments with dedup
Exactly-once effect At-least-once + dedup None visible, extra storage and care Money, inventory, anything not safe to repeat

A good rule of thumb: build for at-least-once, and make the receiver safe against duplicates. That works even when the network, the broker and your own retry code all misbehave.

Timeouts in practice

Choosing the timeout is a trade-off. Too short and you retry calls that were merely slow, adding load and duplicates. Too long and a dead server makes every caller wait. Good defaults come from measurement: a timeout somewhat above the 99th percentile latency of healthy calls. Always set a timeout; a call with no timeout can hang forever and quietly eat your threads. See resilience patterns for how timeouts combine with circuit breakers.

Message passing, middleware and gRPC

Underneath every protocol in this lesson is message passing: processes share no memory, so they cooperate only by sending messages over channels. The core pieces are a sender, a receiver, a channel (a TCP connection, a UDP socket, a queue) and a message format. A socket is the operating system's endpoint for such a channel. TCP sockets give an ordered, reliable byte stream with acknowledgements and retransmission built in; UDP sockets give single datagrams that may be lost, duplicated or reordered, in exchange for lower latency and no connection setup. Receivers use buffering: incoming messages wait in a queue until the application reads them, and when the buffer fills up, either the sender is slowed down (flow control) or messages are dropped.

Three delivery shapes appear again and again:

  • Unicast: one sender, one receiver (an RPC call).
  • Multicast: one sender, a chosen group (replicas of a shard, subscribers of a topic).
  • Broadcast: one sender, every node (cluster membership news, shutdown signals). Broadcast is expensive and is usually built from unicasts, or from gossip when the cluster is large.

Message passing is loosely coupled and scales well, but it is harder to reason about: delivery may be delayed or lost, and sender and receiver need an agreed format.

Middleware

Middleware is the software layer between the operating system and your application that hides these details and offers a uniform programming model. Its core jobs are hiding differences between machines and languages, serializing data (turning objects into bytes and back), finding peers, and providing shared services such as authentication, transactions and logging. Common families:

  • RPC middleware (gRPC, Thrift): call a remote function as if it were local.
  • Message-oriented middleware (Kafka, RabbitMQ): asynchronous queues and topics, covered in Queues, Pub/Sub and Streams.
  • Object middleware (CORBA, Java RMI): remote objects with identity.
  • Database middleware (drivers, connection pools, proxies): uniform access to different data stores.

Middleware is not free: it adds latency, a version-compatibility burden (old clients and new servers must still understand each other, which is why serialization formats allow adding optional fields), a larger attack surface, and one more thing to operate.

The sketch below models a bus with unicast, multicast and broadcast, and a middleware hook that serializes every message.

Python Unicast, multicast, broadcast and a serialization hook

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

gRPC in a nutshell

gRPC is a widely used RPC framework. You describe services and messages in a Protocol Buffers .proto file, and a code generator produces client stubs and server skeletons in many languages, so a call looks like a local method call. It runs over HTTP/2 (multiplexed streams on one connection) and encodes messages in compact binary. It supports four call types:

Type Client sends Server sends Example
Unary one message one message GetUser
Server streaming one message a stream subscribe to price updates
Client streaming a stream one message upload chunks, get a summary
Bidirectional streaming a stream a stream chat, live collaboration

Errors are not free text: every call ends with a status code such as OK, NOT_FOUND, UNAUTHENTICATED, UNAVAILABLE or DEADLINE_EXCEEDED, plus an optional message. Clients set a deadline (an absolute point in time by which the whole call must finish), and the deadline travels with the call so downstream services stop work that nobody is waiting for. Interceptors are hooks that wrap every call on the client or server, the natural place for logging, authentication and metrics. Transport security uses TLS, and per-call credentials (tokens) are attached as metadata.

Python Deadlines, status codes and interceptors (a stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This is a stand-in written for the browser: real gRPC needs generated stubs and a network. It shows the shape: interceptors wrap the handler from the outside in, and a slow handler turns into DEADLINE_EXCEEDED. For the messaging side of the same idea, see Queues, Pub/Sub and Streams.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A client's request times out. What can it safely conclude?

    A timeout only means no reply arrived in time. The request may be lost, the server may be slow or dead, or the reply may be lost after the work was done.
  2. 2. Which guarantee can produce duplicate executions?

    At-least-once retries until acknowledged. If the reply is lost, the retry makes the server execute the operation again.
  3. 3. How do real systems achieve 'exactly-once' behavior?

    Exactly-once delivery is impossible over a lossy network. Exactly-once effect comes from retries combined with deduplication on the receiver.
  4. 4. Which workload fits at-most-once best?

    When a lost message costs little but a duplicate or a retry storm would cost more, fire-and-forget at-most-once is fine.

Summary

  • Nodes communicate only by messages, and the network can lose, delay, duplicate and reorder them.
  • Request/response (RPC) waits for an answer; messaging sends one way and moves on.
  • A timeout is a guess: it cannot tell a lost request from a lost reply or a slow server.
  • At-most-once never retries (messages may be lost); at-least-once retries (duplicates possible).
  • Exactly-once delivery is a myth; exactly-once effect is at-least-once plus deduplication with unique request IDs.
  • Always set timeouts, and build receivers that are safe against duplicates.

Next: Idempotency, Retries and Backoff