Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 36 ยท Building Blocks

Distributed Locks, Leases and Fencing

Why distributed locks are hard: lease expiry, GC pauses and stale clients. Learn fencing tokens and run a safe lock service simulation in your browser.

In a single program, a lock is easy: one thread holds it, the others wait, and when the thread finishes it lets go. In a distributed system, the "threads" are separate machines that can crash, freeze or lose their network connection while holding the lock. Nobody can tell whether the holder is dead, slow, or simply unreachable. If you wait for it forever, everything stops. If you take the lock away, two machines might work on the same resource at once.

This lesson explains why distributed locks are tricky, how leases make them survive crashes, why a lease alone is still not safe, and how fencing tokens close the gap. It builds directly on leader election: a leader is just a lock with a long lease.

Why a plain lock is dangerous

Suppose a client takes a lock in a shared service, does some work and then releases it. What if the client crashes in the middle? The lock stays held forever and everyone else waits. A lock that survives crashes needs a way to be taken back.

That brings us to the idea of a lease.

Leases: locks with an expiry date

A lease is a lock that is valid only for a fixed time, such as 10 seconds. The holder must finish the work or renew the lease before it expires. If the holder dies, the lease simply runs out and somebody else can take over. No one needs to know whether the old holder is really dead.

Lock versus lease

Plain lock Held until released. A crash blocks everyone.
Lease Held for a limited time. A crash heals itself.
Renewal A live holder extends the lease before it ends.

Leases trade a risk of blocking for a risk of overlap.

1 / 2

Leases fix the crash problem and introduce a new one: the holder and the lock service disagree about whether the lease is still valid. The service believes it expired. The holder has not noticed, because it was paused and its own clock stopped caring. Time on different machines can drift (see physical clocks, drift and skew), and a process can be frozen by a long garbage-collection pause, a page fault storm, a VM migration or a laptop lid closing.

The dangerous timeline looks like this:

sequenceDiagram
    participant A as Client A
    participant L as Lock service
    participant B as Client B
    participant S as Storage
    A->>L: acquire lease
    L-->>A: granted until t=10
    A->>S: write line 1
    Note over A: A freezes for a long pause
    Note over L: lease expires at t=10
    B->>L: acquire lease
    L-->>B: granted
    B->>S: write
    Note over A: A wakes up at t=20
    A->>S: stale write line 2

Client A's second write arrives after B has taken over, yet A still believes it holds the lock.

You cannot fix this by making A check the lease right before writing. A can be paused between the check and the write, so the check proves nothing. The only party that can tell the write is stale is the one that sees both writers: the storage.

Fencing tokens

A fencing token is a number that grows each time the lock is granted. The lock service hands the token to the client along with the lease. The client passes it with every request to the protected resource. The resource remembers the highest token it has seen and rejects any request with a smaller one.

With this rule, a stale client is harmless. Its token is older than the new holder's token, so the storage turns the write away, whatever the clocks say. This is the same technique as in leader election: the term number is the token.

A lock service simulation

The simulation below uses virtual time. The lock service grants a lease of 10 seconds. Client A takes it, writes, and then freezes for a long time. Client B takes the lease after it expires. We run the same story twice: first with no tokens, then with the storage checking them.

Python A stale write with and without fencing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In the first run all three writes are accepted, and the final file mixes data from A and B in the wrong order. In the second run A's token is 1, B's is 2, and the storage rejects A's late write. The file contains only valid writes, even though A never found out that it had lost the lock.

What if the storage can't check tokens?

Fencing needs support from the resource. Databases can do it with a version column and a conditional update (UPDATE ... WHERE version = ?), object stores with conditional puts, and some file systems with generation numbers. If the resource cannot reject stale writes, a lease alone cannot make the work safe. In that case the lock only reduces the chance of overlap. That can be acceptable for efficiency (avoiding duplicate work), but not for correctness.

Let's try this in a second simulation. It uses a version-style check on a counter: two workers each read the value and write back an increment, and the second one has a stale view.

Python Conditional writes as fencing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both workers read the value 0. Worker 1 writes first and succeeds, so the version changes. Worker 2's write is refused because it was based on an old version. After re-reading, it succeeds, and the value is 2. Without the check, one increment would have been lost.

Semaphores, barriers and deadlock

A lock admits one holder. A counting semaphore admits up to N: at most N workers may talk to a rate-limited API, run a heavy job, or hold a connection at once. A binary semaphore (N = 1) behaves like a lock. In a distributed system the permit count lives in a coordination service, and each permit should be a lease that expires, for exactly the reason leases exist for locks: a crashed holder must not keep a permit forever.

A barrier makes a group wait until every member has arrived before any may continue. It is the tool for phased work: all mappers finish before any reducer starts, or all nodes load a configuration version before traffic is switched. Real barriers also need a timeout and a way to handle a member that never arrives (otherwise one crash stalls everyone), and are usually built from a counter or a set of ephemeral entries in a store such as ZooKeeper or etcd.

Python A lease-based semaphore and a barrier

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Deadlock appears when clients hold some resources while waiting for others, in a cycle: A holds lock 1 and wants lock 2, B holds lock 2 and wants lock 1. Neither can ever proceed. Across machines nobody sees the whole picture, so deadlocks are harder to detect. Practical defenses: always acquire multiple locks in one global order, use leases and timeouts so a stuck holder eventually loses its claim, acquire all resources at once or none, and keep critical sections small. Detection algorithms build a distributed wait-for graph and look for a cycle, but timeouts are the usual choice in practice.

Choosing a lock service

  • Use a consensus-backed store. etcd, ZooKeeper and Consul keep locks and leases in a replicated log (see Raft), so the lock service itself survives crashes. ZooKeeper's zxid and etcd's revision numbers work well as fencing tokens.
  • Be careful with a single Redis instance. A single node loses its locks if it fails, and a failover can hand the same lock to a second client. Multi-node schemes exist, but they depend on timing assumptions and don't produce fencing tokens by themselves.
  • Keep leases short but renewable, and renew in a background loop, so a crashed holder is replaced quickly.
  • Prefer idempotent work. If duplicate work is harmless (see idempotency and retries), you may not need a lock for correctness at all.
  • Use monotonic clocks for measuring local lease time, not wall clocks, which can jump.

Ask yourself: "If two clients both believe they hold the lock for a few seconds, what is the worst that can happen?" If the answer is "some wasted work", a lease is fine. If it is "corrupted money", you need a fencing token or a conditional write.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why do distributed locks usually use leases instead of plain locks?

    Nobody can tell whether a silent holder is dead or slow. An expiry time lets the system recover from a crashed holder without that knowledge.
  2. 2. A client checks that its lease is valid and then writes to storage, but the write is still stale. How is that possible?

    Any pause, such as a garbage-collection stop, can fall between the check and the write, so the check cannot guarantee safety.
  3. 3. What does a storage system do with a fencing token?

    The storage sees both writers and can refuse the stale one, which makes safety independent of clocks and pauses.
  4. 4. Which is true about using a lock only for efficiency, such as avoiding duplicate work?

    If overlap is harmless, correctness does not depend on the lock. Fencing is for cases where overlap would corrupt data.

Summary

  • A lock held by a machine that may crash or freeze needs an expiry, so distributed locks are normally leases.
  • A lease alone cannot prevent a paused client from acting after it expired.
  • A fencing token (an increasing number) lets the protected resource reject stale clients; conditional writes with a version number do the same job.
  • Use consensus-backed services such as etcd or ZooKeeper for locks, keep local timing on monotonic clocks, and ask whether you need the lock for correctness or only for efficiency.

Next: Distributed Caching.