Skip to content
Tutorials Center
My learning

Topics

Lesson 50 of 66 ยท Files and Storage

The Unix File System, Inodes and Journaling

Explore the Unix file system layout (superblock, inodes, block groups), file types, hard and soft links, file permissions and how journaling keeps a crash from corrupting data.

The Unix family (Linux, macOS, the BSDs, Android) shares a way of organizing storage that has barely changed in fifty years. Its central character is the inode. Learn how the inode, the directory, the superblock and the journal fit together, and you understand ext4, which most Linux systems use, and you have the vocabulary for APFS, XFS and NTFS too.

Everything is a file

Unix's famous design rule is that devices, pipes and sockets also appear in the file tree. The first character of ls -l shows the type:

Symbol Type Example
- Ordinary (regular) file /home/ana/notes.txt
d Directory /home
l Symbolic link /usr/bin/python
c Character device /dev/tty (bytes in a stream)
b Block device /dev/sda (random-access blocks)
p Named pipe (FIFO) created with mkfifo
s Socket local network endpoints

A program can read and write a keyboard, a disk and a file with the same two calls.

The on-disk layout

A Unix file system divides the volume into regions:

  • Boot block: code to start the OS (if the volume is bootable).
  • Superblock: global facts: block size, number of inodes and blocks, counts of free ones, a "clean/dirty" flag.
  • Inode table: a fixed array of inodes, one per file.
  • Data blocks: the contents of files and directories.

ext4 splits the disk into block groups, each with its own copy of the free-space bitmaps, a slice of the inode table and its data blocks. Keeping a file's inode and data in the same group shortens seeks on a hard disk.

flowchart LR
    B["Boot block"] --> S["Superblock"]
    S --> G1["Block group 0<br/>bitmaps, inodes, data"]
    G1 --> G2["Block group 1<br/>bitmaps, inodes, data"]
    G2 --> G3["..."]

The superblock describes the volume; each block group is a small file system of its own.

The inode

An inode (index node) stores everything about a file except its name and, of course, its contents:

  • type and permission bits, owner (UID) and group (GID),
  • size in bytes,
  • timestamps (access, modify, and inode-change),
  • the link count,
  • pointers to data blocks: 12 direct, then single, double and triple indirect (or extents in ext4).

With 4 KB blocks and 4-byte pointers, one indirect block holds 1,024 pointers. That gives:

  • direct: 12 x 4 KB = 48 KB
  • single indirect: 1,024 x 4 KB = 4 MB
  • double indirect: 1,024 x 1,024 x 4 KB = 4 GB
  • triple indirect: 1,024^3 x 4 KB = 4 TB
Python How far can one inode reach?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Small files, the majority on any system, are reached with one read, and only huge files pay for the extra indirection.

Directories and path lookup

A directory is a file whose data is a list of (name, inode number) pairs. Opening /home/ana/a.txt follows a chain:

  1. Read the root inode (number 2, by convention) and its data: find home.
  2. Read inode of home and its data: find ana.
  3. Read inode of ana and its data: find a.txt.
  4. Read inode of a.txt. Permissions are checked at each directory (execute bit) and at the file.

The OS caches directory entries (the dentry cache) and inodes, so repeated lookups cost no disk reads.

A hard link is another (name, inode) pair, so it increases the inode's link count (see directory structures). A symbolic link is an inode of type l whose data is a path string, resolved on every access.

Permissions are nine bits: read/write/execute for owner, group and others, written as rwxr-xr-- or octal 754. For directories, r means you can list names, w means you can add and remove entries, and x means you can pass through to reach things inside. Special bits are setuid (run with the owner's rights), setgid and the sticky bit (only the owner may delete files in the directory, as in /tmp). The next chapter, protection and access control, covers the idea more broadly.

Python Decode Unix permission bits

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Why crashes are dangerous

Creating a file touches several structures: the inode bitmap (claim an inode), the inode itself (initialize it), the directory (add the name), and the data blocks and block bitmap if the file has contents. The disk can do only one write at a time, so a power cut after step 2 but before step 3 leaves an allocated inode that no directory names (a leak). Other orders produce worse results: a name that points to an uninitialized inode, or two files sharing one block.

Before journaling, the fix was fsck (file system check): after an unclean shutdown, scan the whole disk, cross-check bitmaps, inodes and directories, and repair. It takes minutes to hours on big disks.

Journaling

A journal (write-ahead log) is a small reserved area. Before changing the main structures, the file system writes a description of the whole change as one transaction into the journal, then writes a commit record. Only after the commit does it apply (checkpoint) the changes to their real places.

After a crash, recovery reads the journal:

  • A transaction with a commit record is replayed (re-applied), so the change is complete.
  • A transaction without a commit record is discarded, so the change never happened.

Either way the file system is consistent, and recovery takes seconds.

Python A write-ahead journal surviving a crash

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both crash cases end in a consistent state: either the file exists with everything it needs, or it does not exist at all.

Journal modes

ext3 and ext4 offer three levels:

Mode What goes in the journal Safety Speed
journal Metadata and file data Highest Slowest (data written twice)
ordered (default) Metadata only, but data blocks are written first Good Good
writeback Metadata only, no ordering for data Metadata safe, data may be stale Fastest

Journaling protects the file system structures, not necessarily your last few seconds of data. An application that needs a file to be safe calls fsync. Other designs reach the same goal differently: copy-on-write file systems (ZFS, Btrfs, APFS) never overwrite live data, and log-structured systems write everything sequentially.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which information is NOT stored in a Unix inode?

    Names live in directory entries that map a name to an inode number. This is also why one inode can have several names.
  2. 2. With 4 KB blocks and 4-byte pointers, how many data blocks does a single indirect block reach?

    4096 / 4 = 1,024 pointers per indirect block, so it addresses 1,024 data blocks (4 MB).
  3. 3. After a crash, how does a journaling file system treat a transaction with no commit record?

    Without a commit record the transaction was incomplete, so ignoring it leaves the structures as they were before.
  4. 4. What does the execute bit mean on a directory?

    Execute (search) permission lets you pass through the directory. Listing needs read permission.

Summary

  • Unix treats files, directories, links, devices, pipes and sockets uniformly; ls -l shows the type.
  • The superblock describes the volume, block groups hold bitmaps, inodes and data, and each inode holds attributes plus direct and indirect pointers (or extents) but not the name.
  • Directories map names to inode numbers; hard links add names, symbolic links store a path.
  • Permissions are rwx for owner, group and others, with setuid, setgid and sticky bits.
  • Journaling writes a transaction and a commit record before applying changes, so recovery replays committed work and drops the rest. Copy-on-write designs are an alternative.

Next: I/O Hardware, Interrupts and DMA