Skip to content
Tutorials Center
My learning

Topics

Lesson 93 of 95 ยท System Administration

Rescue Mode and Recovery

Recover a Linux system that will not boot or let you in: reset a forgotten password, fix a broken fstab, repair a filesystem with fsck, and use chroot from a live USB.

Sooner or later a Linux machine stops cooperating. You forget the only password. A typo in /etc/fstab drops the boot into a text prompt that says "emergency mode". A power cut leaves a filesystem damaged. A firewall rule locks you out of your own server. None of this is a disaster if you know the small set of tools designed for it: rescue and emergency modes, a live USB or provider rescue image, chroot, and fsck.

This lesson is about your own machines, practiced on virtual machines where a mistake costs nothing. Most recovery steps need a console, root rights and sometimes a reboot, so they cannot run in the browser editors on this page. The commands that touch the real system are read-only blocks. The logic behind them is ordinary text processing, so we rebuild the key parts as runnable Bash: a linter for fstab, a decoder for fsck exit codes, and a safe experiment with the /etc/shadow file format.

The rules of recovery

Before any specific case, a few habits prevent a bad day from becoming a worse one:

  • Change one thing at a time, and copy a file before editing it: cp /etc/fstab /etc/fstab.bak.
  • Keep a second session open while you change SSH, firewall or PAM settings. If the change locks the new sessions out, the old one is your way back in.
  • Snapshot first. On a VM or cloud instance a snapshot takes seconds and turns a risky repair into a safe one.
  • Read the error. Emergency mode tells you which unit failed. journalctl -xb shows the log of the current boot, and journalctl -b -1 the previous one.
  • Never run fsck on a mounted filesystem, and never guess with -y on a disk that might be failing. Copy what you can first.
flowchart TD
  A["Something is wrong"] --> B{"Can you log in at all?"}
  B -- "yes" --> C["Fix it as an admin:<br/>sudo, journalctl, systemctl"]
  B -- "no, but a text shell appears" --> D["Read the error in emergency mode:<br/>fstab, filesystem or disk"]
  B -- "no, nothing starts" --> E["Boot a live USB or rescue image,<br/>mount the system and chroot"]
  D --> F["Remount root read-write,<br/>fix the file, reboot"]
  E --> F

Pick the lightest tool that gets you a shell: normal sudo, then emergency mode, then a live image.

Forgotten password, but you have sudo

If you only forgot your password and another administrator (or root) can still log in, this is trivial:

CONSOLE
$ sudo passwd ada            # set a new password for ada
$ sudo chage -d 0 ada        # force ada to choose a new one at the next login
$ sudo passwd -S ada         # status: P = usable password, L = locked
$ sudo passwd -l ada         # lock the account;  passwd -u unlocks it

On Ubuntu the root account has no password by default (it is locked), and your first user gets root rights through sudo. See Users, Groups and sudo for the setup.

Nobody can log in: rescue and emergency mode

If every administrator password is lost, you need access to the machine's console (a physical keyboard, a hypervisor window, or a cloud provider's web console). Anyone with that access and an unencrypted disk can become root. That is not a bug: it is why server rooms have locks and why laptops should use full-disk encryption (LUKS), which stops this method completely.

Systemd offers two special boot targets:

Target What you get
rescue.target Basic system up, local filesystems mounted, a root shell ("single-user mode").
emergency.target Almost nothing running; root filesystem mounted read-only. Use it when even rescue fails.

You pick one at the boot menu. On a typical GRUB-based system:

  1. Hold Shift (BIOS) or tap Esc (UEFI) during boot to show the GRUB menu.
  2. Highlight your entry and press e to edit it for this one boot.
  3. Find the line that starts with linux and add to its end: systemd.unit=rescue.target.
  4. Press Ctrl-X to boot.

Many systems then ask for the root password (through sulogin). If root has none, or you do not know it, replace that kernel argument with init=/bin/bash, which starts a bare shell instead of the init system. Ubuntu also has a ready-made menu entry under Advanced options, called recovery mode, with a "root" item that drops to a shell.

In that bare shell the root filesystem is mounted read-only, so remount it before changing anything:

CONSOLE
# mount -o remount,rw /
# passwd ada                 # choose the new password
# sync                       # flush disk buffers
# exec /sbin/init            # continue booting normally  (or: reboot -f)

On Red Hat family systems (RHEL, Rocky, Fedora) the method is rd.break on the kernel line. It stops in the initramfs, with the real root mounted at /sysroot:

CONSOLE
# mount -o remount,rw /sysroot
# chroot /sysroot
# passwd root
# touch /.autorelabel        # SELinux must relabel the changed file on the next boot
# exit
# exit

The .autorelabel step is easy to forget, and without it SELinux can refuse the login that you just fixed.

What a password reset really changes

Passwords are never stored as text. /etc/shadow holds one line per account, with colon-separated fields; the second field is a salted hash. passwd replaces that field. The login program hashes what you type with the salt that is stored in the field and compares the result. A leading ! on the hash locks the account: nothing can ever hash to a value that starts with !.

We can do all of this with a fabricated shadow file, which is exactly how you would test a repair script before touching the real one:

Bash The shadow file format: check, reset and lock

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

On a real system you never edit /etc/shadow by hand: use passwd, chpasswd or vipw -s, which lock the file while you work. The point of the demo is to see that a reset is a one-field change and that a lock is a one-character prefix.

Boot from a live USB and chroot

When the boot loader itself is damaged, or you need a full working environment, boot a live USB (any distribution's installer image) or your cloud provider's rescue mode. Then you mount the broken system's disk and use chroot to "enter" it, so commands act on that installation as if you had booted it.

sequenceDiagram
  participant L as Live system
  participant D as Broken installation
  L->>D: mount the root partition on /mnt
  L->>D: bind-mount /dev, /proc, /sys into /mnt
  L->>D: chroot /mnt
  Note over D: commands now act on the installed system
  D->>D: passwd, edit fstab, update-grub
  L->>D: exit, umount, reboot

A live system borrows the kernel; chroot makes the installed system's files look like the root of the machine.

CONSOLE
# lsblk -f                                   # find the partitions and their filesystems
# mount /dev/sda2 /mnt                       # the root partition (yours may differ)
# mount /dev/sda1 /mnt/boot/efi              # the EFI partition, on UEFI machines
# for d in dev proc sys run; do mount --rbind /$d /mnt/$d; done
# chroot /mnt /bin/bash
(chroot)# passwd ada
(chroot)# update-grub                        # Debian/Ubuntu; Fedora family: grub2-mkconfig -o <grub.cfg path>
(chroot)# grub-install /dev/sda              # reinstall the boot loader (BIOS machines)
(chroot)# exit
# umount -R /mnt
# reboot

If the disk is encrypted, open it first with cryptsetup open /dev/sda3 cryptroot; with LVM, run vgchange -ay so the logical volumes appear. The bind mounts matter: without /proc and /dev inside the chroot, tools like update-grub fail in confusing ways.

A broken /etc/fstab

/etc/fstab lists the filesystems to mount at boot. If an entry names a disk that does not exist, a mount point that is missing or has a typo, systemd waits and then gives up, and the machine boots into emergency mode with a message such as "You are in emergency mode" and a request for the root password. The repair is short once you have a shell:

CONSOLE
# mount -o remount,rw /          # the root filesystem is read-only in emergency mode
# nano /etc/fstab                # fix or comment out the bad line
# mount -a                       # try every entry now; errors show up here
# findmnt --verify               # lint the file (util-linux 2.29 and newer)
# systemctl daemon-reload
# reboot

Each line has six fields: device, mount point, type, options, dump flag and fsck pass. Use UUIDs (from blkid or lsblk -f) rather than names like /dev/sdb1, which can change from boot to boot. For disks that may be absent (a USB drive, a network share), add nofail so a missing device does not stop the boot, and x-systemd.device-timeout=10s so you do not wait ninety seconds for it.

Because this file is plain text, we can write a checker. The sandbox gets two fixtures: an fstab and a list of the UUIDs that blkid reports.

Bash A linter for /etc/fstab

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Notice what the checker cannot tell you: whether the type matches the real filesystem, or whether the mount point directory exists. The real mount -a and findmnt --verify know more, and are the final test before every reboot.

Checking and repairing a filesystem with fsck

After a crash or a power cut, the kernel can usually replay the journal of ext4 and XFS and continue. When it cannot, or when you see "structure needs cleaning" or I/O errors, run a filesystem check on an unmounted filesystem: from emergency mode, from a live USB, or after umount.

CONSOLE
# umount /dev/sdb1                 # never check a mounted filesystem
# fsck -n /dev/sdb1                # -n: look only, answer "no" to every repair question
# fsck -f /dev/sdb1                # force a full check even if the filesystem looks clean
# fsck -p /dev/sdb1                # -p: fix only what is safe, without questions
# xfs_repair -n /dev/sdc1          # XFS has its own tool (-n = dry run); fsck.xfs does nothing

fsck is a front end that starts the right program for the type, usually e2fsck for ext2/3/4. Files that were recovered but lost their names appear in the lost+found directory at the top of that filesystem, named by inode number. If the disk itself is failing (smartctl -a /dev/sdb reports reallocated sectors or dmesg shows I/O errors), do not repair it in place: copy it first with ddrescue to a good disk and repair the copy.

The exit status of fsck is a bit mask, so one number can carry several facts. Scripts and boot logic read it like this:

Bit Value Meaning
0 1 Errors were found and corrected
1 2 The system should be rebooted
2 4 Errors were left uncorrected
3 8 Operational error
4 16 Usage or syntax error
5 32 Cancelled by the user
7 128 Shared-library error

A status of 3 therefore means "corrected errors, and you must reboot". A decoder is one loop with a bit test:

Bash Decoding fsck exit codes with bit tests

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

A boot script treats 0 and 1 as success. Status 4 or higher means a human has to look at the disk before the filesystem is mounted again. See Disks, Mounts, Logs and Monitoring for how to read the disks involved.

Locked out of a remote server

You rarely have a keyboard attached to a cloud server, but you do have other doors:

  • The provider's web console or serial console gives a login prompt even when the network or SSH is broken. This needs a working local password, so set one up before you need it, or use the provider's rescue mode instead.
  • Rescue mode boots a small system from the provider's image and attaches your disk. You mount it, fix the problem (a bad sshd_config, wrong permissions on ~/.ssh/authorized_keys, a firewall rule that blocks port 22), then reboot into the normal system.
  • As a last resort, detach the volume, attach it to a second instance, mount it there and edit the files, exactly as in the chroot section.

Before reloading SSH after any edit, test the file: sudo sshd -t prints nothing if the syntax is valid. And remember the rule about keeping a second session open, from SSH, scp and Working Remotely.

A full disk can also lock you out in odd ways: logins fail because no file can be written. In rescue mode check df -h (space) and df -i (inodes), find the culprits with du -xh --max-depth=1 / | sort -h, trim the journal with journalctl --vacuum-size=200M, and look for deleted files that a process still holds open with lsof +L1.

Prevention is cheaper

  • Take snapshots or backups before upgrades and before touching fstab, sshd_config, firewall rules or the boot loader.
  • Test changes with the system's own checkers (mount -a, sshd -t, visudo -c, nginx -t) before rebooting or reloading.
  • Keep an out-of-band way in: console access, a second admin account, or a documented rescue procedure.
  • Put /etc under version control with a tool such as etckeeper, so every change is a recorded diff.
  • Store the recovery steps for each machine type where you can reach them when the machine is down, not on the machine itself.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. You booted with init=/bin/bash to reset a password, but passwd says the filesystem is read-only. What do you do first?

    In rescue shells the root filesystem is mounted read-only. Remounting it read-write lets passwd write /etc/shadow.
  2. 2. Which line in /etc/fstab helps a machine boot even if a USB drive is unplugged?

    nofail tells systemd that a missing device is not fatal for the boot. Combine it with x-systemd.device-timeout so the boot does not wait long.
  3. 3. fsck exits with status 4. What does that mean?

    The status is a bit mask: 1 corrected, 2 reboot required, 4 errors left uncorrected. A status of 4 needs manual attention before the filesystem is used again.
  4. 4. Why do you bind-mount /dev, /proc and /sys before running chroot from a live USB?

    A chroot only shows the files under the new root. The installed system's /dev, /proc and /sys directories are empty, so you make the live system's versions visible inside it.
  5. 5. What does a ! at the start of the hash field in /etc/shadow do?

    passwd -l simply prefixes the hash with !. The password is kept so passwd -u can restore it, but no input can match that field.

Summary

  • Work calmly: copy files before editing, change one thing at a time, keep a second session open and snapshot VMs first.
  • Forgotten password with sudo available: sudo passwd user. With no access at all: edit the GRUB entry (rescue.target, init=/bin/bash, or rd.break on Red Hat systems), remount / read-write, run passwd, and relabel for SELinux if needed.
  • For a damaged system or boot loader, boot a live USB or provider rescue image, mount the root partition, bind-mount /dev, /proc, /sys, and chroot.
  • A bad /etc/fstab causes emergency mode; fix it, test with mount -a and findmnt --verify, and use UUIDs plus nofail for removable disks.
  • Run fsck only on unmounted filesystems, start with -n, and read its exit status as a bit mask; use xfs_repair for XFS and ddrescue before repairing failing hardware.
  • Disk encryption, backups and an out-of-band console are what make recovery safe, and what stop others from doing it to you.

Next: Automation with Ansible and Terraform