Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 47 ยท HDFS

Filesystems Hadoop Can Use

Learn Hadoop's FileSystem abstraction and its implementations: local, HDFS, WebHDFS, S3A, ViewFs, Azure and Google Cloud connectors, and how URI schemes pick one.

HDFS is the best-known Hadoop file system, but Hadoop does not depend on it. Every tool in the ecosystem (MapReduce, Hive, Spark, Pig) reads and writes data through one Java abstraction, org.apache.hadoop.fs.FileSystem, and a URI scheme at the start of a path decides which implementation does the work. That is why the same distcp command can copy from HDFS to S3, and why a Spark job runs unchanged against local files, HDFS or a cloud bucket.

This lesson explains the abstraction, the main implementations, their differences (especially for object stores), and how to choose. Hadoop cannot run on this site, so the commands are read-only; the runnable example is a Python stand-in that mimics scheme-based dispatch. It follows Working with HDFS from the Shell.

One interface, many back ends

A path such as hdfs://namenode:8020/data/logs/part-0000 has three parts: the scheme (hdfs), the authority (namenode:8020) and the path (/data/logs/part-0000). Hadoop looks up the scheme in its configuration, finds the class registered for it (property fs.<scheme>.impl) and creates a FileSystem object for that authority. Your code asks that object to open, create, listStatus, rename and delete; it never needs to know whether bytes come from a disk, a cluster or a web service.

If you write a path without a scheme (/data/logs), Hadoop uses the default file system, fs.defaultFS in core-site.xml. That setting is why hdfs dfs -ls / on a cluster lists HDFS while the same command in standalone mode lists your local disk (see Installing Hadoop).

Terminal
hadoop fs -ls file:///tmp                     # local disk
hadoop fs -ls hdfs://nn1:8020/data            # an explicit HDFS cluster
hadoop fs -ls /data                           # whatever fs.defaultFS says
hadoop fs -cp /data/clicks s3a://analytics-archive/clicks

The common implementations

Scheme Implementation What it is
file:// LocalFileSystem the local disk, used in standalone mode and for tests
hdfs:// DistributedFileSystem HDFS, talking to the NameNode and DataNodes with a native protocol
webhdfs://, swebhdfs:// WebHdfsFileSystem HDFS over an HTTP REST API (swebhdfs uses HTTPS), useful across firewalls and from non-Java clients
viewfs:// ViewFileSystem a client-side mount table that stitches several HDFS namespaces into one tree (federation)
s3a:// S3AFileSystem Amazon S3 and compatible object stores
abfs://, abfss:// AzureBlobFileSystem Azure Data Lake Storage Gen2 (abfss is the TLS variant)
gs:// GoogleHadoopFileSystem Google Cloud Storage connector
har:// HarFileSystem Hadoop archives: many small files packed into one archive, readable as a file system
ofs://, o3fs:// Ozone FileSystem Apache Ozone, an object store with an HDFS-like interface

The old s3:// and s3n:// schemes belong to earlier connectors; use s3a:// today. Azure's older wasb:// connector is also legacy.

WebHDFS

HDFS clients normally need the Hadoop libraries and direct access to many DataNode ports. WebHDFS exposes the same operations as REST calls (GET /webhdfs/v1/data/file?op=OPEN, op=LISTSTATUS, op=CREATE) served by the NameNode, which redirects the data transfer to a DataNode. Any tool that can speak HTTP (curl, Python, a browser) can then read and write HDFS, and distcp can copy between clusters of different versions using webhdfs:// because it avoids binary protocol differences. It is slower than the native protocol, so use it for access convenience, not bulk throughput.

ViewFs

With HDFS federation a cluster has several NameNodes, each owning part of the namespace (see NameNode HA, Federation and Operations). ViewFs lets clients see one tree: a mount table in core-site.xml maps /user to one NameNode and /data/logs to another. Applications use viewfs://cluster/user/ana and never learn which NameNode serves it.

XML
<property>
  <name>fs.viewfs.mounttable.cluster.link./user</name>
  <value>hdfs://nn1:8020/user</value>
</property>
<property>
  <name>fs.viewfs.mounttable.cluster.link./data</name>
  <value>hdfs://nn2:8020/data</value>
</property>

Object stores are not file systems

S3A, ABFS and GCS make buckets look like directories, but object stores differ from HDFS in important ways, and jobs that ignore this can be slow or wrong:

  • Rename is not cheap. On HDFS rename is an atomic metadata change. On an object store it is copy then delete, taking time proportional to the data size and not atomic. The classic MapReduce output committer promotes task output by renaming, which is both slow and unsafe on S3, so use the S3A committers (staging or magic) or a table format that commits by metadata (Iceberg, Delta).
  • Directories are an illusion. There are only keys with / in their names; an "empty directory" is a marker object.
  • Listing is slower and consistency rules differ by store. Amazon S3 has been strongly consistent for reads after writes and listings since late 2020, but other S3-compatible stores may not be.
  • No data locality. HDFS lets YARN run computation where the data is. A bucket is remote, so every byte crosses the network, and tuning shifts to connection pools, block sizes for multipart upload and caching.
  • Permissions and append work differently, and appending to an existing object is generally unsupported.

None of this makes object stores bad: they are cheap, durable and elastic, which is why many modern lakes keep the data in a bucket and use Hadoop-compatible engines for processing (see Data Lakes, Warehouses and Lakehouses).

Python Scheme-based dispatch and the cost of rename

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Choosing a file system

Use HDFS for hot, frequently processed data on a cluster that has local disks, where locality and cheap atomic renames help. Use object stores for durable, elastic, shared storage and cold data, with committers or table formats. Use WebHDFS when clients cannot use the native protocol, and ViewFs to hide federation. Use file:// only for tests. Moving data between them is the everyday job of distcp (see Working with HDFS from the Shell), for example hadoop distcp hdfs://nn1/data s3a://bucket/data.

The connectors need the right JARs on the classpath (for S3A, hadoop-aws and the AWS SDK) and credentials. Prefer instance roles or credential providers over putting secrets in core-site.xml, and never in job code (see Security, Governance and Data Quality).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How does Hadoop decide which FileSystem class handles a path such as s3a://bucket/data?

    The scheme (s3a) is looked up in the configuration (fs.s3a.impl) and creates the matching FileSystem object.
  2. 2. What happens when you use a path with no scheme, like /data/logs?

    fs.defaultFS in core-site.xml supplies the default scheme and authority.
  3. 3. Why is renaming a directory slow on S3A?

    Object stores have no atomic rename, which is why object-store committers or table formats are used.
  4. 4. What is ViewFs used for?

    A mount table maps paths to different HDFS namespaces so applications see a single hierarchy.

Summary

  • Every Hadoop tool uses the FileSystem abstraction; the URI scheme selects the implementation, and a path without a scheme uses fs.defaultFS.
  • Main schemes: file, hdfs, webhdfs, viewfs, s3a, abfs, gs, har and Ozone's ofs.
  • WebHDFS offers a REST interface to HDFS; ViewFs merges federated namespaces on the client.
  • Object stores lack atomic rename, real directories and data locality; use S3A committers or table formats and expect network-bound performance.
  • distcp copies between file systems, for example HDFS to S3.

Next: NameNode HA, Federation and Operations.