Skip to content
Tutorials Center
My learning

Topics

Lesson 39 of 47 ยท Architecture and Operations

Sizing, Monitoring and Running a Cluster

Plan Hadoop cluster capacity, watch the right metrics, recognise common failures and compare on-premises with cloud costs, with a capacity calculator.

Sooner or later someone asks the practical question: "How many machines do we need, and how will we know when something is wrong?" Sizing a cluster is mostly arithmetic once you know which numbers matter. Running it is mostly noticing small problems (a disk, a slow node, a growing pile of tiny files) before they become outages.

This lesson covers capacity planning for storage and compute, the tools and metrics for monitoring, the failures you will meet most often, and the cloud versus on-premises decision. As usual the real tools (Ambari, Cloudera Manager, Prometheus, the Hadoop web UIs) need a cluster, so their commands are read-only. The runnable examples are a capacity calculator and a health-check rule engine.

Every number below (disk sizes, prices, ratios) is an example chosen to make the arithmetic clear. Replace them with your own measurements before you buy anything.

Capacity planning in four questions

Plan storage first

Daily intake How much new data arrives per day?
Compression Columnar plus Snappy or ZSTD shrinks it.
Replication Three copies means three times the disk.
Headroom Reserve space for temp files and growth.
1 / 2

Storage

Start with the raw data per day and work up:

  1. Compress. Raw text or JSON compresses well, and a columnar format such as Parquet compresses it even more (see Row vs Column Formats). Measure a real sample: a ratio of 3x to 5x is common, but your data may differ.
  2. Multiply by retention. Daily size times the days you keep (or the number of years of history), plus growth. A business growing 50% a year needs next year's capacity bought this year.
  3. Replicate. With replication factor 3 (see Replication and Block Placement) every byte uses three bytes of disk. Erasure coding such as RS-6-3 uses 1.5x for cold data.
  4. Add working space. Intermediate job output, spill files and the output of ETL need room. A typical allowance is 25 to 40% of the disk, and you should never run HDFS above about 80 to 85% full: performance and the balancer both suffer.

Divide the result by the usable disk per DataNode, and you have a node count for storage.

Compute

Compute sizing starts from the workload, not the data. Each YARN container asks for memory and cores (see YARN: Resources and Scheduling). A worker with 256 GB of RAM, of which about 200 GB is given to YARN, holds roughly 50 containers of 4 GB. If your peak workload needs 4,000 concurrent containers, that is about 80 workers. Cores must keep up (about one core per container is a starting point for CPU-heavy jobs), and a 10 Gb network is the common minimum because shuffle and re-replication move large volumes.

Typical node types differ:

  • Worker nodes (DataNode plus NodeManager): many disks (12 to 24 spinning drives), plenty of RAM.
  • Master nodes (NameNode, ResourceManager, journal nodes, ZooKeeper): few but reliable, with lots of RAM for the NameNode metadata (see NameNode HA, Federation and Operations), and redundant disks.
  • Edge nodes: where users log in and submit jobs, so they never touch the masters.

A capacity calculator

The stand-in below turns the checklist into code. Change the numbers to see how sensitive the answer is to replication, compression and headroom.

Python Storage and node-count calculator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The final design needs the larger of the two node counts, and then some margin, because a cluster that can barely hold today's workload fails during a node loss. Plan for losing at least a rack's worth of capacity (and the re-replication traffic that follows).

Monitoring: tools and metrics

Hadoop comes with web UIs (the NameNode on port 9870 and the ResourceManager on port 8088 in Hadoop 3) and JMX metrics exposed over HTTP. Management suites wrap those in dashboards: Apache Ambari (now retired but still found in older estates) and Cloudera Manager install, configure, restart and monitor services across all nodes with rolling upgrades. Many teams export metrics to Prometheus and Grafana and send alerts through their usual paging system. A few useful read-only commands:

Terminal
hdfs dfsadmin -report          # capacity, used, live and dead DataNodes
hdfs fsck / -blocks            # missing, corrupt, under-replicated blocks
yarn node -list -all           # NodeManager states
yarn application -list         # running applications
curl http://nn1:9870/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem

Which numbers deserve an alert?

  • HDFS: missing and corrupt blocks (should be zero), under-replicated blocks (should be draining), capacity used above about 80%, dead DataNodes, NameNode heap use and RPC queue time, the number of files and blocks.
  • YARN: pending applications and pending memory (jobs waiting for resources), failed NodeManagers, container failures, queue usage against the queue's limit.
  • Nodes: disk failures, full local disks (YARN stops using a node when its disks go over about 90% full), CPU steal, swap, network errors.
  • Jobs: duration compared with the usual duration for the same job. A pipeline that normally takes 20 minutes and now takes three hours is an incident even if it succeeds.

Common failures and what they mean

Symptom Usual cause First response
Many under-replicated blocks A DataNode or rack was lost Let re-replication run; check why the node died
Missing blocks All replicas gone (multiple failures) Restore the node; otherwise restore from backup or distcp copy
NameNode slow or out of memory Too many small files Merge files, raise heap, consider federation
Jobs stuck in ACCEPTED No free resources or queue at its limit Check queue usage and preemption settings
Containers killed for memory Container too small for the task Raise the container memory and keep the Java heap about 80% of it
One node is always slow Failing disk or noisy neighbour Compare metrics; decommission it
HDFS in safe mode NameNode starting, or too few blocks reported Wait for block reports; check missing DataNodes

The "missing blocks" row is a reminder that backups still matter. Replication protects against a dead disk, not against a bad rm -r. Turn on HDFS trash and snapshots, and copy critical data to another cluster or a cloud bucket with distcp.

Here is a small stand-in for a monitoring system: a list of metrics with rules, producing alerts in order of severity.

Python A health-check rule engine

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real alerting adds two ideas: a condition must hold for several minutes before it fires (to avoid flapping), and each alert links to a runbook that says what to do.

Cloud or on-premises?

On-premises means buying servers, a data center, power and staff, with a low cost per terabyte if the cluster stays busy. The cloud means renting: you pay by the hour, can scale in minutes and avoid hardware work, but a cluster that runs 24/7 costs more per unit.

The big architectural difference is separating storage from compute. In the cloud, data usually sits in an object store (S3, GCS, Azure storage) that costs little and lives independently, and you start a temporary Spark or Hive cluster, for example on EMR, Dataproc or HDInsight, only when you need it. On-premises HDFS ties the disks to the compute nodes (data locality), which is fast but means you must buy compute to keep storage.

Questions that decide it:

  • Is the workload steady (favors owned hardware) or spiky (favors cloud)?
  • Do you have the people to run a cluster, or would managed services save more than they cost?
  • Where must the data legally live, and how often does it move (cloud egress is billed)?
  • How fast must you grow? Ordering hardware takes weeks.

A rough break-even calculation is a good start. The stand-in below compares monthly cost for a steady and a spiky workload using invented prices.

Python Steady versus spiky workloads

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This model ignores storage, egress, discounts for reserved instances, and the cost of staff. Treat its lesson as the shape of the decision: a fixed monthly cost versus a cost that follows usage.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. You need to store 100 TB of compressed data with replication factor 3. About how much raw disk is needed before headroom?

    Each byte is stored three times, so 100 TB of data needs 300 TB of disk. Add working space and keep HDFS below about 80% full on top of that.
  2. 2. Which HDFS metric should normally be exactly zero?

    A missing block means every replica is gone, which is data loss. Under-replicated blocks are common briefly after a failure and should shrink over time.
  3. 3. A NameNode runs out of heap although total data is moderate. What is the most likely cause?

    The NameNode holds metadata for every file and block in memory, so millions of tiny files exhaust its heap regardless of total bytes.
  4. 4. Why is separating storage from compute attractive in the cloud?

    Storage and compute scale and are billed independently, so spiky workloads pay only for the hours they use, at the cost of giving up local-disk speed.

Summary

  • Storage sizing is: daily data, compression, retention and growth, replication, working space and a maximum fill of about 80%.
  • Compute sizing starts from container memory and cores for the peak workload; plan spare capacity for node and rack loss.
  • Master, worker and edge nodes have different hardware needs.
  • Watch missing and under-replicated blocks, disk fullness, NameNode heap, pending YARN memory and job durations.
  • Most failures trace back to dead nodes, small files, undersized containers or full disks; backups, trash and snapshots protect against human error.
  • Cloud versus on-premises is a trade between fixed and usage-based costs, helped by separating storage from compute.

Next: Project: Web Log Analysis with MapReduce.