Skip to content
Tutorials Center
My learning

Topics

Lesson 4 of 36 ยท Time and Ordering

Physical Clocks, Drift and Skew

Understand wall clocks versus monotonic clocks, clock drift, NTP and last-write-wins hazards, with Python simulations of drifting node clocks.

Every computer has a clock, and it is tempting to use it for everything: ordering events, expiring caches, deciding which of two writes is newer. On one machine that works well enough. Across several machines it quietly produces wrong answers, because no two clocks ever agree exactly and no clock is exactly right.

This lesson explains what the hardware actually does, why clocks drift apart, how synchronization narrows (but never closes) the gap, and what goes wrong when software trusts timestamps from different machines. The next two lessons then show how to order events without trusting physical time.

Two kinds of clock

Operating systems expose two different clocks, and mixing them up is a classic bug.

Wall clock vs monotonic clock

Wall clock Date and time of day. Can jump forwards or backwards.
Monotonic clock Only counts up. Good for measuring durations.
Rule of thumb Intervals use monotonic. Dates use wall.

Neither is comparable between two machines.

1 / 2

A time-of-day (wall) clock returns the current date and time, for example "seconds since 1970". It is synchronized with outside time sources, so it can be stepped: if the machine was fast, the sync may set it back, and the same instant appears twice. Leap seconds, daylight-saving mistakes and a user fiddling with settings cause jumps too.

A monotonic clock counts from some arbitrary starting point (often boot time) and never goes backwards. It is the right tool for measuring how long something took or for timeouts. Its absolute value means nothing, and the value from one machine is meaningless on another.

Python
import time

start = time.monotonic()     # right for "how long did this take?"
# ... do some work ...
elapsed = time.monotonic() - start

now = time.time()            # wall clock, right for "what date is it?"

If you measure a duration with the wall clock and the clock is stepped back during the measurement, you can get a negative duration. Many outages have started that way.

Why clocks drift

A computer clock counts the ticks of a quartz crystal. Crystals are cheap and slightly imprecise; their frequency also changes with temperature. A typical error is tens of parts per million (ppm). Twenty ppm sounds tiny, but it is 20 microseconds of error per second, which is roughly 1.7 seconds per day. That is called clock drift: the rate at which a clock gains or loses time.

The difference between two clocks at one moment is clock skew. If node A reads 12:00:00.000 and node B reads 12:00:00.250 at the same real instant, the skew is 250 ms.

Simulating drift

Let us make five nodes with different drift rates and see how far apart they get, first if nobody corrects them, then with a periodic correction. The correction mimics a time-sync protocol that asks a better clock for the time and adjusts, with a small remaining error.

Python Drifting clocks with and without synchronization

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without synchronization the nodes drift apart by seconds within a day. Syncing hourly still leaves hundreds of milliseconds (about 330 here), and syncing every minute gets it down to roughly 15 ms. That is the best case: the sync itself has errors, because the request and reply take unequal, unknown times to cross the network.

Keeping clocks close: the idea of NTP

The Network Time Protocol (NTP) is how most machines discipline their clocks. The idea is simple. The client sends a request stamped t1, the server notes when it received it (t2) and when it replied (t3), and the client notes when the reply arrived (t4). Assuming the trip out and back took equal time, the client estimates its offset as:

offset โ‰ˆ ((t2 - t1) + (t3 - t4)) / 2

The assumption of symmetric delay is exactly where the error sneaks in: if the outbound path was slower than the return path, the estimate is wrong by half the difference. Over the public Internet, expect errors of a few to tens of milliseconds; inside a well-run data center with good local time sources, microseconds to a millisecond. Some cloud providers and companies run special hardware (GPS or atomic clocks) to get tighter bounds, and a few databases expose time as an interval of uncertainty instead of a single number, then wait out the uncertainty before committing. That is an advanced technique, but it shows the right attitude: a timestamp is an estimate with error bars.

Synchronization algorithms: Cristian, Berkeley and NTP layers

NTP is one member of a family. The older algorithms are short enough to read in full and show the trade-offs.

Cristian's algorithm assumes one trusted time server. The client records its own time before sending a request (t0), receives the server's time, and records the time on arrival (t1). The round trip is rtt = t1 - t0, and the client sets its clock to server_time + rtt / 2. If the real one-way delays differ, the answer is off, but never by more than rtt / 2, so the client even knows its error bound. Taking several samples and keeping the one with the smallest round trip tightens the bound.

The Berkeley algorithm has no trusted time source. A coordinator asks every node for its clock, averages all the readings (ignoring wild outliers), and tells each node how much to adjust. It sends adjustments, not absolute times, because an absolute time would be stale by the time it arrives. This keeps the group consistent with itself, even if it is not correct with respect to real time.

Python Cristian's estimate and Berkeley averaging

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With equal delays Cristian's algorithm is exact (the correct adjustment is -250 ms). With the outbound trip three times slower it is off by 20 ms, inside the reported bound of 40 ms.

NTP organizes servers into strata. Stratum 0 devices are reference clocks (GPS receivers, atomic clocks). Stratum 1 servers are connected directly to them, stratum 2 servers sync from stratum 1, and so on; each layer adds a little error. NTP does not jump the clock when the error is small: it slews, running the clock slightly faster or slower until it catches up, so time never goes backwards. Very large errors are corrected with a step. Where microseconds matter (finance, telecoms, industrial control), the Precision Time Protocol uses hardware timestamping in network cards and switches to remove most software delay.

Some systems go further and expose the uncertainty itself. Google's TrueTime API returns an interval [earliest, latest] instead of a single number, backed by GPS and atomic clocks in every data center. A transaction picks a commit timestamp and then waits out the uncertainty before reporting success, so any later transaction anywhere is guaranteed a larger timestamp. The cost is a commit delay of a few milliseconds; the benefit is globally consistent ordering from physical time.

Why timestamps lie

Now the main hazard. A very common design is last write wins (LWW): when two writes to the same key conflict, keep the one with the later timestamp. It is simple, needs no coordination and is used in several databases. But the timestamp comes from whichever node accepted the write.

Python Last write wins with skewed clocks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The later write was silently discarded. Both replicas agree and no error is reported, yet the user's correction vanished. The skew was only 300 ms, but a gap between writes smaller than the skew is enough. In a busy system, with skew of tens of milliseconds and writes arriving in bursts, this happens regularly.

Other timestamp hazards:

  • Ties. Two writes in the same millisecond; the loser is arbitrary.
  • Clock jumps. A node whose clock jumps far into the future "wins" every conflict until real time catches up, and its stale data can overwrite everything.
  • Leases and expiry. A process thinks its lease is valid because its clock says so while the lock service's clock says otherwise (see locks and leases).
  • Log merging. Sorting log lines from several machines by timestamp can show an effect before its cause.

Never use wall-clock timestamps from different machines to decide which event happened first or to resolve conflicts, unless you accept losing some writes. Use them for human-readable logs, rough ordering and expiry with generous margins.

What to do instead

You have several options, each with trade-offs:

  1. Single source of order. Let one node assign sequence numbers (a leader), as in leader-based replication. This costs a round trip and a bottleneck, but order is exact.
  2. Logical clocks. Count events instead of seconds. Lamport clocks give a consistent order, and vector clocks detect when two events were truly concurrent.
  3. Tight, bounded physical clocks. Specialized hardware and uncertainty intervals, for systems that can afford them.
  4. Conflict-free designs. Data types such as CRDTs merge concurrent updates without needing to know which came first.

Choose the weakest tool that satisfies the requirement. Physical time is fine when an error of a few seconds does not matter, such as session expiry or cache TTLs, and dangerous exactly when it decides correctness.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which clock should you use to measure how long a request took?

    A wall clock can be stepped by time sync, giving odd or negative durations. A monotonic clock only moves forward, so differences are reliable on the same machine.
  2. 2. A crystal clock drifts by 20 ppm. Roughly how far off is it after one day?

    20 millionths of 86,400 seconds is about 1.7 seconds. This is why clocks need continuous correction.
  3. 3. What does the NTP offset estimate assume that can be wrong?

    The offset formula halves the round trip, which is exact only if both directions took the same time. Any asymmetry becomes an error in the estimate.
  4. 4. Why can last-write-wins with wall-clock timestamps lose a user's newer write?

    Skew between nodes means the "later" write can receive an earlier timestamp and then lose the comparison, silently.
  5. 5. Which use of wall-clock time is generally acceptable?

    Coarse expiry tolerates errors of seconds. The other uses need exact ordering, which skewed wall clocks cannot provide.

Summary

  • Wall clocks give dates and can jump; monotonic clocks measure durations and never go backwards. Neither is comparable between machines.
  • Crystals drift by tens of ppm, which is seconds per day; synchronization like NTP narrows skew to milliseconds, never zero.
  • A timestamp is an estimate with error bars.
  • Last-write-wins on physical timestamps can silently discard the newer write when clocks are skewed.
  • For ordering, prefer a single sequencer, logical clocks, bounded-uncertainty clocks or conflict-free data types.

Next: Lamport Clocks and Happens-Before, a way to order events by cause instead of by the wall.