Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 47 ยท HDFS

Working with HDFS from the Shell

Learn the everyday HDFS shell commands: ls, put, get, cat, rm, permissions, quotas, fsck and distcp, with a runnable Python mini file system to practice on.

Most of the time you meet HDFS through its command line. The hdfs dfs command looks like the Linux file commands you already know (ls, cat, rm, mkdir), but every path lives in the cluster's file system instead of on your laptop's disk. Learning a dozen commands lets you load data, inspect it, clean up and check the health of the cluster.

This lesson shows those commands, permissions and quotas, the fsck health tool and distcp for copying between clusters. If shell basics are new to you, read the Linux tutorial first. For what happens behind each command, see The HDFS Read and Write Path.

The real hdfs command needs a Hadoop cluster, so the shell blocks below are read-only. To let you practice, the runnable examples contain a small Python mini file system that mimics several commands on an in-memory tree. It is a stand-in: it shows the same ideas and similar output, but it has no blocks, DataNodes or NameNode.

The shape of an HDFS command

Every command follows hdfs dfs -<command> [options] <paths>. The older spelling hadoop fs also works. Paths are either full URIs such as hdfs://namenode:8020/data/logs or, more commonly, plain absolute paths such as /data/logs, which use the default file system from core-site.xml (fs.defaultFS). A relative path means "inside your home directory", /user/<your-name>.

Terminal
hdfs dfs -help            # list every command
hdfs dfs -help ls         # help for one command

Everyday commands

Terminal
# create directories (-p creates parents, like Linux)
hdfs dfs -mkdir -p /user/ana/logs

# copy a local file INTO the cluster
hdfs dfs -put access.log /user/ana/logs/

# list a directory (-h = human sizes, -R = recursive)
hdfs dfs -ls -h /user/ana/logs

# print a file, or only its end
hdfs dfs -cat /user/ana/logs/access.log | head
hdfs dfs -tail /user/ana/logs/access.log

# copy a file OUT of the cluster
hdfs dfs -get /user/ana/logs/access.log ./copy.log

# copy and move inside the cluster (metadata only for -mv)
hdfs dfs -cp /user/ana/logs/access.log /user/ana/backup.log
hdfs dfs -mv /user/ana/backup.log /user/ana/old.log

# sizes
hdfs dfs -du -h /user/ana        # size per entry
hdfs dfs -df -h /                # capacity of the cluster

# delete
hdfs dfs -rm /user/ana/old.log
hdfs dfs -rm -r /user/ana/logs   # directory

A few details worth remembering:

  • -put and -copyFromLocal are nearly the same; -get and -copyToLocal likewise. -moveFromLocal deletes the local file after upload.
  • -mv inside HDFS only changes the namespace in the NameNode, so moving a 10 TB directory is instant. -cp really copies the data.
  • -appendToFile adds to the end of an existing file. HDFS cannot edit in the middle of a file: it follows a write-once, append-only model. To "change" a file you rewrite it.
  • -getmerge dir out.txt concatenates many small part files (such as MapReduce output) into one local file.
  • -text prints compressed or SequenceFile data as text, which -cat cannot.

The trash

With trash enabled (fs.trash.interval, in minutes), -rm does not delete: it moves the file to /user/<you>/.Trash, and a background process purges it after the interval. -rm -skipTrash deletes immediately and permanently. This has saved many careers; on production clusters, leave trash on.

More HDFS commands

Beyond the basics, a second group of commands covers creating empty files, moving files in, inspecting metadata, changing replication and managing access control lists.

Terminal
hdfs dfs -touchz /data/flags/_READY               # create an empty (zero-length) file, often used as a marker
hdfs dfs -put -f report.csv /data/reports/        # -f overwrites an existing target (plain -put refuses)
hdfs dfs -put -p report.csv /data/reports/        # keep timestamps, ownership and permissions
hdfs dfs -moveFromLocal big.log /data/logs/       # upload and then delete the local file
hdfs dfs -copyFromLocal a.txt /data/              # like -put, local source only
hdfs dfs -copyToLocal /data/a.txt ./              # like -get, local destination only
hdfs dfs -appendToFile more.txt /data/a.txt       # append local data to an existing file
hdfs dfs -getmerge /data/out/part-* merged.txt    # concatenate many HDFS files into ONE local file
hdfs dfs -text /data/events.seq                   # print a sequence file or compressed file as text
hdfs dfs -checksum /data/a.txt                    # show the file checksum
hdfs dfs -count -q -h /data/logs                  # directories, files, bytes and quota information
hdfs dfs -test -e /data/_SUCCESS && echo ready    # -e exists, -d directory, -z zero length (exit code only)

Inspecting a file with -stat. It prints chosen properties using format letters: %n name, %b size in bytes, %o block size, %r replication, %u owner, %y modification time, %F file or directory type.

Terminal
hdfs dfs -stat "%n %b bytes, block %o, replicas %r" /data/a.txt
# a.txt 5120 bytes, block 134217728, replicas 3

Disk usage. -du -h shows sizes (and, with -v, headers); -du -s summarizes a directory in one line, which replaces the old -dus command (deprecated). Recent versions show both the file size and the space consumed with all replicas.

Changing replication. -setrep changes the target replication of existing files; -R applies it to a whole directory tree and -w waits until the change completes:

Terminal
hdfs dfs -setrep -w 2 /data/old-logs             # keep only two copies of old data
hdfs dfs -setrep -R 1 /tmp/scratch               # scratch data needs no safety copies

Lowering replication frees space immediately as excess replicas are deleted; raising it makes the NameNode schedule copies in the background. Replication of a file is part of its metadata (see Replication and Block Placement).

Access control lists (ACLs). Basic user, group and other permissions cannot express "also let the analytics team read this". ACLs (enabled with dfs.namenode.acls.enabled=true) add named users and groups:

Terminal
hdfs dfs -setfacl -m user:maria:r-x /data/sales            # modify: give maria read and execute
hdfs dfs -setfacl -m group:analytics:r-x /data/sales
hdfs dfs -setfacl -m default:group:analytics:r-x /data/sales   # default ACL: inherited by NEW files in the directory
hdfs dfs -getfacl /data/sales                              # show the ACL
hdfs dfs -setfacl -x user:maria /data/sales                # remove one entry
hdfs dfs -setfacl -b /data/sales                           # remove all extended entries

A + at the end of the permission string in -ls output (drwxr-x---+) means a file or directory has an ACL. As with all HDFS permissions, they only mean something when Kerberos authentication is on (see Security, Governance and Data Quality).

Python getmerge, setrep and stat on a tiny model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Permissions

HDFS uses the same model as POSIX: an owner, a group and nine permission bits, with r, w and x. For a file x means nothing, and for a directory x lets you enter it. Commands match Linux:

Terminal
hdfs dfs -chmod 640 /user/ana/logs/access.log
hdfs dfs -chown ana:analysts /user/ana/logs
hdfs dfs -chmod -R 750 /user/ana
hdfs dfs -getfacl /user/ana/logs        # ACLs, if enabled

Without Kerberos, HDFS trusts the user name the client claims, so permissions are a safety net against mistakes, not real security. Real authentication comes later, in Security, Governance and Data Quality. The HDFS superuser is whoever started the NameNode (often hdfs).

Quotas

Administrators can stop one team from filling the cluster with two kinds of quota on a directory:

  • Name quota: maximum number of files and directories (protects the NameNode's memory).
  • Space quota: maximum bytes, counted with replication. A 1 GB file with replication 3 uses 3 GB of quota.
Terminal
hdfs dfsadmin -setQuota 100000 /user/ana        # at most 100,000 names
hdfs dfsadmin -setSpaceQuota 2t /user/ana       # 2 TB of raw space
hdfs dfs -count -q -h /user/ana                 # show quotas and usage
hdfs dfsadmin -clrQuota /user/ana               # remove

Writing past a quota fails with NSQuotaExceededException or DSQuotaExceededException.

Checking health with fsck

hdfs fsck walks the namespace and reports on blocks. It does not repair anything (the NameNode re-replicates by itself); it tells you what the NameNode knows.

Terminal
hdfs fsck /                                   # whole-cluster summary
hdfs fsck /data/events -files -blocks -locations
hdfs fsck / -list-corruptfileblocks

The summary ends with lines like Under-replicated blocks, Mis-replicated blocks, Corrupt blocks, Missing replicas and finally The filesystem under path '/' is HEALTHY or CORRUPT. A file is corrupt when every replica of one of its blocks is lost. -files -blocks -locations shows which DataNodes hold each block, which is how you learn why a file reads slowly or what a failed node held.

Copying between clusters: distcp

hadoop distcp copies huge directory trees between clusters (or between HDFS and cloud storage) by running a MapReduce job whose map tasks do the copying in parallel. There is no reduce step.

Terminal
hadoop distcp hdfs://prod-nn:8020/data/events hdfs://backup-nn:8020/data/events
hadoop distcp -update -delete -m 50 /data/events s3a://my-bucket/events
  • -update copies only files that are missing or different, so repeated runs are incremental.
  • -delete removes files at the target that no longer exist at the source (use with -update).
  • -m 50 caps the number of parallel map tasks.
  • -p preserves permissions, owner and replication.

Commands to remember

put / get Move files in and out.
ls / cat / du Look at what is there.
fsck Check blocks and health.
distcp Parallel copy between clusters.

Everything is append-only; rewriting a file is the only way to change it.

1 / 2

Practice with a mini file system

The stand-in below keeps a tree of directories and files in a Python dictionary and implements mkdir -p, put, ls, cat, du and rm -r. It also splits each file into 16-byte "blocks" so that du can show raw usage with replication, the same arithmetic quotas use.

Python A mini hdfs dfs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The second stand-in imitates -count -q: it enforces a name quota and a space quota when files are added, failing the way HDFS does.

Python Name and space quotas

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which command copies a local file into HDFS?

    -put (or -copyFromLocal) uploads a local file; -get downloads one.
  2. 2. Why is hdfs dfs -mv of a huge directory almost instant?

    Paths live in the NameNode's namespace. Renaming updates that metadata and leaves the blocks where they are.
  3. 3. A directory has a 1 GB space quota. How large a file with replication 3 can you store at most?

    Space quotas count raw bytes including replicas, so 1 GB of quota holds about 1/3 GB of logical data at replication 3.
  4. 4. What does hdfs fsck / do when it finds a under-replicated block?

    fsck only reports. Re-replication is a NameNode background task.

Summary

  • hdfs dfs -<command> mirrors Linux file commands: mkdir, put, get, ls, cat, cp, mv, du, rm.
  • Files are append-only; -mv is metadata-only, -cp copies data, and trash protects against accidental deletes.
  • Permissions follow POSIX but are only trustworthy with Kerberos.
  • Name and space quotas limit directories; space is counted with replication.
  • hdfs fsck reports file, block and replica health without repairing.
  • distcp copies large trees in parallel with map-only MapReduce jobs; -update makes it incremental.

Next: NameNode HA, Federation and Operations.