Skip to content
Tutorials Center
My learning

Topics

Lesson 2 of 47 ยท Big Data Basics

Distributed Systems in Plain Words

Learn the ideas under every big data tool: nodes and networks, partial failure, replication, consistency, the CAP idea and why commodity hardware works.

A distributed system is a group of computers that cooperate so that, to the user, they look like one system. Hadoop, Kafka, HBase and Spark are all distributed systems, and they all face the same few problems. If you understand those problems now, the design choices in the rest of the tutorial will feel natural instead of arbitrary.

This lesson uses plain words and small simulations. You will learn what nodes and networks are, why partial failure is the central difficulty, how replication protects data, what consistency means, and what the CAP idea says. It follows What Is Big Data?.

This is a concept lesson. Nothing here needs a cluster: the two runnable examples are stand-ins written in Python that simulate nodes with ordinary objects. Real systems do the same things with real machines and a real network.

Nodes and the network

A node is one machine (or one virtual machine) taking part in the system. A group of nodes that work together is a cluster. Nodes talk to each other over a network, and the network is the part that makes distributed systems hard.

On one computer, calling a function either returns or crashes, and you know which. Between two computers, you send a message and then... wait. If no answer comes, you cannot tell whether

  • the message was lost,
  • the other node is down,
  • the other node is just slow,
  • or it did the work and the reply was lost.

This uncertainty is built into the physics and cannot be removed, only managed. The usual tool is the timeout: wait a while, then assume the node is dead. Timeouts can be wrong, so good systems make operations safe to repeat.

Partial failure

On a single machine, things mostly work or everything is down. In a cluster, some parts fail while others keep running. That is partial failure, and it is guaranteed to happen at scale. If one disk fails on average once every few years, then with 1,000 disks you will lose a disk about every day or two.

Things that fail in a cluster

Disks The most common failure of all.
Nodes Power, memory or kernel problems.
Network Switches die and links get slow.
Stragglers Nodes that are alive but very slow.

Plan for failure as a normal event, not a surprise.

1 / 2

The key mindset of Hadoop is: failure is normal, so the software handles it, not the hardware. That is why Hadoop can run on cheap commodity hardware, meaning standard, inexpensive servers rather than special, fault-tolerant machines. Buying reliability in hardware is very expensive. Buying many cheap machines and letting software replace broken ones is much cheaper.

Replication

The standard defence against losing a node is replication: keep several copies of every piece of data on different nodes. If one copy disappears, another still exists, and the system can make a new copy to get back to the wanted number.

The next stand-in simulates five nodes. It stores each block on three of them, then "kills" nodes one by one and checks whether every block is still readable. The placement here is deliberately simple (round-robin); Replication and Block Placement shows how HDFS really chooses.

Python How many node failures can replication survive?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With three copies, any two simultaneous node failures are survivable. A third failure can lose the blocks whose three copies happened to sit on exactly the three dead nodes. The lesson: the replication factor is a trade between safety and disk space, and replicas on different nodes (and later, different racks) matter more than the number alone.

Replication also helps speed: many readers can use different copies at the same time.

Consistency: do all copies agree?

Copies create a new problem. If you write a new value, there is a moment when some replicas have it and others do not. What should a reader see?

  • Strong consistency: every reader sees the latest write, as if there were one copy. Simple to reason about, but replicas must coordinate before answering, which costs time and may block during failures.
  • Eventual consistency: replicas catch up eventually. A reader may briefly see an old value, but the system stays fast and available.

Neither is "right". A bank balance wants strong consistency. A "likes" counter on a video can usually be a little behind.

HDFS avoids much of the trouble by design: files are written once and then only read or appended, so there are no concurrent edits to reconcile (HDFS Architecture explains this).

The CAP idea

The CAP theorem is a famous summary of the trade-off. It talks about three properties:

  • Consistency (C): reads see the latest write.
  • Availability (A): every request to a working node gets an answer.
  • Partition tolerance (P): the system keeps working when the network splits the nodes into groups that cannot talk to each other.

Networks do split, so P is not optional in a real cluster. The theorem says that while a partition is happening, you must choose: either refuse some requests to stay consistent (CP), or answer every request but risk stale or conflicting data (AP). When there is no partition, you can have both, though there is still a trade between consistency and latency.

flowchart TD
  P["Network partition happens"] --> Q{"What do you favor?"}
  Q -->|"Consistency"| CP["CP: some requests wait or fail"]
  Q -->|"Availability"| AP["AP: answer, maybe stale data"]

During a partition a system picks consistency or availability; it cannot give both.

Systems lean in different directions: HBase and ZooKeeper favour consistency, while Cassandra and Dynamo-style stores are tuned for availability (Choosing a NoSQL Store). Treat CAP as a way to ask the right question about a tool, not as a label that settles everything.

A failure-detecting heartbeat

How does a cluster notice a dead node? Each node regularly sends a small "I'm alive" message called a heartbeat. A coordinator marks a node dead when it has heard nothing for too long. HDFS does exactly this (you will see it in The HDFS Read and Write Path).

This stand-in runs a clock in whole "seconds". Nodes send heartbeats every second, except that node n3 crashes at second 4 and n4 is only slow. The coordinator declares a node dead after 3 silent seconds.

Python Detecting a dead node with heartbeats

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Only n3 is declared dead, at second 6 (it was last heard at second 3). The slow node n4 survives because its beats still arrive often enough. Choosing the timeout is a real trade: too short and healthy but busy nodes are declared dead (causing needless copying), too long and real failures go unnoticed.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is 'partial failure'?

    In a cluster, nodes, disks or links fail independently, so the system must keep working with some parts missing.
  2. 2. Why can Hadoop run on cheap commodity hardware?

    Hadoop puts fault tolerance in software through replication and retries, which is far cheaper than fault-tolerant hardware.
  3. 3. A network partition happens. According to the CAP idea, what must a system choose?

    Partitions cannot be avoided, so during one the system either stays consistent by refusing some requests, or stays available and may return stale data.
  4. 4. With a replication factor of 3, how many simultaneous node failures is every block guaranteed to survive?

    Each block has three copies on three different nodes. Two failures leave at least one copy, but three could remove all copies of some block.

Summary

  • A cluster is many nodes cooperating over a network; the network makes everything uncertain.
  • Partial failure is normal at scale, so software must handle it. This lets Hadoop use commodity hardware.
  • Replication keeps several copies on different nodes and also helps read speed.
  • Consistency describes whether readers see the latest write; strong and eventual are the two ends.
  • CAP: during a network partition you choose between consistency and availability.
  • Heartbeats and timeouts detect dead nodes; the timeout is a trade-off.

Next: The Hadoop Ecosystem at a Glance.