Skip to content
Tutorials Center
My learning

Topics

Lesson 47 of 47 ยท Projects, Practice and Review

Hadoop and Big Data Cheat Sheet

A one-page Hadoop and big data cheat sheet: HDFS, YARN and Hive commands, key defaults, file format comparison and a table for choosing the right tool.

This is the page to keep open while you work. It collects the commands, defaults, formats and tool choices from the whole tutorial in compact tables. Nothing here is new; each section links back to the lesson with the explanation.

The commands are read-only here: this site cannot run Hadoop. Two small Python stand-ins at the end let you try the "which tool?" and "how many blocks?" logic. Defaults are the usual Hadoop 3 values; your cluster's *-site.xml files may differ.

What is on this sheet

Commands HDFS, YARN, Hive.
Defaults Numbers worth knowing.
Formats Which file for which job.
Tool choice Problem to tool.
1 / 2

HDFS shell commands

More detail in Working with HDFS from the Shell.

Task Command
List a directory hdfs dfs -ls /data
Make directories hdfs dfs -mkdir -p /data/raw/2026
Upload a local file hdfs dfs -put local.csv /data/raw/
Download hdfs dfs -get /data/raw/local.csv .
Print a file hdfs dfs -cat /data/raw/local.csv
Last kilobyte hdfs dfs -tail /data/raw/local.csv
Copy / move inside HDFS hdfs dfs -cp src dst, hdfs dfs -mv src dst
Delete (recursive) hdfs dfs -rm -r /data/tmp
Skip the trash hdfs dfs -rm -r -skipTrash /data/tmp
Sizes, human readable hdfs dfs -du -h /data
Free space hdfs dfs -df -h
Change replication hdfs dfs -setrep -w 2 /data/raw/local.csv
Permissions and owner hdfs dfs -chmod 750 /data/raw, hdfs dfs -chown etl:analytics /data/raw
Quota usage hdfs dfs -count -q -h /data
Merge many files into one local file hdfs dfs -getmerge /data/out merged.txt

HDFS administration

See NameNode HA, Federation and Operations.

Task Command
Check file health and block locations hdfs fsck /data -files -blocks -locations
Cluster report (live and dead nodes) hdfs dfsadmin -report
Safe mode status hdfs dfsadmin -safemode get
Rebalance DataNodes hdfs balancer -threshold 10
Copy between clusters in parallel hadoop distcp hdfs://a/data hdfs://b/data
HA state of a NameNode hdfs haadmin -getServiceState nn1

YARN and MapReduce commands

See YARN: Resources and Scheduling and Writing a MapReduce Job in Java.

Task Command
Run a job hadoop jar app.jar MainClass /in /out
Run a streaming job hadoop jar hadoop-streaming-*.jar -input /in -output /out -mapper m.py -reducer r.py -file m.py -file r.py
List running applications yarn application -list
Kill an application yarn application -kill application_1700000000000_0001
Fetch the logs of a finished app yarn logs -applicationId application_1700000000000_0001
List nodes yarn node -list -all
Show queues yarn queue -status default

Hive and Beeline

See Hive: SQL over Files.

SQL
-- connect: beeline -u jdbc:hive2://host:10000 -n analyst
SHOW DATABASES;
SHOW TABLES;
DESCRIBE FORMATTED sales;           -- location, format, partitions, table type
SHOW PARTITIONS sales;
MSCK REPAIR TABLE sales;            -- register partition folders added outside Hive
EXPLAIN SELECT country, SUM(amount) FROM sales WHERE dt = '2026-03-10' GROUP BY country;

CREATE EXTERNAL TABLE sales (id BIGINT, country STRING, amount DOUBLE)
PARTITIONED BY (dt STRING)
STORED AS PARQUET
LOCATION '/data/curated/sales';

Key defaults and numbers

Setting Usual default Where it matters
dfs.blocksize 128 MB Number of blocks and map tasks
dfs.replication 3 Raw storage = data x 3
Erasure coding RS(6,3) 1.5x overhead Cold data, survives 3 lost blocks per group
NameNode memory per object about 150 bytes Small-files problem
DataNode heartbeat every 3 s Failure detection
DataNode considered dead about 10.5 minutes without heartbeat Re-replication starts
mapreduce.task.io.sort.mb 100 MB Map-side sort buffer
mapreduce.map.sort.spill.percent 0.80 When the buffer spills to disk
mapreduce.job.reduces 1 Choose it yourself
fs.trash.interval 0 (off); often set to 1440 minutes Safety net for -rm
yarn.scheduler.minimum-allocation-mb 1024 Smallest container
NameNode RPC / web UI 8020 / 9870 Client connections, browser
ResourceManager web UI 8088 Application tracking
HiveServer2 10000 Beeline and JDBC

Format comparison

Details in Row vs Column Formats, Avro and Schema Evolution and Parquet and ORC Internals.

Format Layout Splittable Schema Best for
CSV / JSON lines Row, text Yes (uncompressed) None or loose Exchange, debugging
Gzip-compressed text Row, text No None Small archives only
SequenceFile Row, binary Yes Key and value types Merging small files
Avro Row, binary Yes Embedded, evolvable Streaming, ingestion
Parquet Columnar Yes (by row group) Embedded Analytics, Spark, Hive
ORC Columnar Yes (by stripe) Embedded Hive, ACID tables

Compression codecs in short: Snappy is fast with a modest ratio (default choice inside Parquet and ORC), Zstandard gives better ratios at similar speed, gzip compresses well but is slow and not splittable as plain text, bzip2 is splittable but slow.

Choosing a tool

You need Reach for
Cheap, durable storage for huge files HDFS or a cloud object store
Share one cluster between engines YARN
Classic batch over huge data MapReduce (or, today, Spark)
SQL analytics on files Hive, Spark SQL, Trino
Millisecond random reads and writes by key HBase
Move data from a database in batch Sqoop-style jobs, today often Spark/connectors
Collect logs and events Flume, NiFi, Kafka
Durable event log and streaming backbone Kafka
Low-latency stateful streaming Flink or Spark Structured Streaming
Coordination and leader election ZooKeeper
ACID tables on a lake Delta Lake, Iceberg or Hudi

See The Hadoop Ecosystem at a Glance and Data Lakes, Warehouses and Lakehouses for the reasoning.

Two quick stand-ins

The first turns the table above into a lookup; the second does the block arithmetic you need most often.

Python Tool chooser

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Python Blocks, replicas and raw storage

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 3 questions

of 3 answered You scored / 3 ยท perfect!

  1. 1. Which command shows where a file's blocks are stored?

    fsck reports each block of a file and the DataNodes that hold its replicas.
  2. 2. Which of these formats is NOT splittable when stored as plain compressed text?

    A gzip stream must be read from the start, so one mapper has to process the whole file.
  3. 3. What is the usual raw storage needed for 10 TB with default replication?

    Replication factor 3 stores every block three times.

Summary

  • hdfs dfs moves and inspects data, hdfs fsck and dfsadmin check health, yarn manages applications, Beeline runs Hive SQL.
  • Remember the defaults: 128 MB blocks, replication 3, 150 bytes of NameNode memory per object, a 100 MB sort buffer.
  • Columnar formats serve analytics, row formats serve ingestion; gzip text cannot be split.
  • Match the tool to the problem: storage, SQL, key lookups, event log or stateful streaming each have a natural home.

That completes the tutorial. Revisit Practice Exercises with Solutions or Hadoop and Big Data Interview Questions any time.