Skip to content
Tutorials Center
My learning

Topics

Lesson 85 of 95 ยท System Administration

Server Hardening Checklist

A practical Linux server hardening checklist: lock down SSH, add a firewall and fail2ban, automate updates, remove risky setuid files and watch changes with auditd.

A freshly installed server on a public IP address is probed within minutes. Bots try root with common passwords on port 22, scan for old web software and look for any open service with a known weakness. Hardening means reducing what an attacker can reach (the attack surface) and making what remains harder to abuse, so that one mistake does not become a breach. No single setting makes a machine "secure"; the goal is defense in depth, several independent layers so that when one fails, the next still holds.

This lesson is a checklist you can apply to a new server in an afternoon, with the reasoning behind each item: SSH, the firewall, fail2ban, updates, services and file permissions, and finally auditing with auditd so you notice when something changes. It builds on SSH, scp and Working Remotely, PAM, sudoers and Account Policies, SELinux and AppArmor and Packages, Services and systemd. The sandbox has no sshd, fail2ban or auditd, so those tools appear in read-only listings, and the runnable examples implement the checks as scripts you can adapt.

The layers

flowchart LR
    I["Internet"] --> F["1. Firewall<br/>only needed ports"]
    F --> S["2. SSH<br/>keys only, no root"]
    S --> B["3. fail2ban<br/>bans repeated failures"]
    B --> A["4. Accounts<br/>sudo, PAM, least privilege"]
    A --> M["5. Minimal services<br/>+ MAC (SELinux, AppArmor)"]
    M --> U["6. Updates<br/>patched software"]
    U --> L["7. Auditing<br/>auditd, logs, alerts"]

Each layer reduces risk on its own; together they make a single mistake survivable.

Before you change anything, keep an extra root or sudo session open and test every access change from a second terminal. Locking yourself out of a remote machine is the most common hardening mistake.

Step 1: lock down SSH

SSH is usually the only door into a server, so it gets the most attention. The effective settings live in /etc/ssh/sshd_config and files in /etc/ssh/sshd_config.d/:

TEXT
# /etc/ssh/sshd_config.d/10-hardening.conf
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
PermitEmptyPasswords no
MaxAuthTries 3
LoginGraceTime 30
X11Forwarding no
AllowUsers deploy maria
ClientAliveInterval 300
ClientAliveCountMax 2

What each line buys you:

  • PermitRootLogin no: nobody can brute-force the best-known account name. Administrators log in as themselves and use sudo.
  • PasswordAuthentication no: passwords can be guessed, keys cannot. Generate a key pair with ssh-keygen -t ed25519, install the public key with ssh-copy-id, test the key login, and only then turn passwords off.
  • MaxAuthTries, LoginGraceTime: fewer guesses per connection, and a short window to authenticate.
  • AllowUsers (or AllowGroups): an allowlist; anyone not named cannot log in even with a valid password or key.
  • X11Forwarding no and friends: switch off features you do not use.

Changing the port from 22 only reduces log noise; it is not a security control. After editing, validate before reloading: sudo sshd -t checks the syntax and prints nothing on success, sudo sshd -T prints the effective configuration, and sudo systemctl reload ssh (or sshd on RHEL) applies it without dropping existing sessions. Then open a new terminal and confirm you can still log in.

One quirk causes surprising audits: in sshd_config the first value found for a keyword wins, not the last. A PasswordAuthentication no appended at the bottom of the main file is silently ignored if an earlier line (or an earlier included file) says yes. The audit script below handles that rule, compares a configuration with a baseline, and uses sshd's own defaults for anything unset:

Bash Audit an sshd_config against a baseline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The audit reports passwordauthentication as yes even though the file later says no; the first line won. That one rule has made many "hardened" servers accept passwords. (The tool that tells the truth on a real machine is sshd -T.) For high-value servers, add a second factor: AuthenticationMethods publickey,keyboard-interactive together with a PAM TOTP module, or put SSH behind a VPN or bastion host.

Step 2: close everything you do not need

A firewall closes ports that no service should expose. Use a default deny policy and open only what you serve:

CONSOLE
# Ubuntu / Debian with ufw
$ sudo ufw default deny incoming
$ sudo ufw default allow outgoing
$ sudo ufw allow OpenSSH
$ sudo ufw allow 443/tcp
$ sudo ufw enable
# RHEL family with firewalld
$ sudo firewall-cmd --permanent --add-service=https
$ sudo firewall-cmd --reload

Then see what is actually listening and remove what should not be there:

CONSOLE
$ sudo ss -tlnp                      # listening TCP sockets and their processes
$ systemctl list-unit-files --state=enabled      # services that start at boot
$ sudo systemctl disable --now cups  # a printer service on a headless server
$ sudo apt purge telnetd             # old plaintext protocols have no place

Databases and caches (MySQL, PostgreSQL, Redis) should listen on 127.0.0.1 or a private network only. A service that is not installed cannot be exploited.

Step 3: fail2ban against brute force

Even with keys only, bots keep knocking and fill your logs. fail2ban reads log files, finds repeated failures with regular expressions (a filter), and bans the source address with the firewall (an action) for a time. Put your settings in /etc/fail2ban/jail.local (never edit jail.conf; updates replace it):

TEXT
# /etc/fail2ban/jail.local
[DEFAULT]
ignoreip = 127.0.0.1/8 ::1 192.0.2.10
bantime  = 1h
findtime = 10m
maxretry = 5

[sshd]
enabled  = true
maxretry = 3

The rule reads: if an address fails 3 times within 10 minutes, ban it for an hour. Add your own office address to ignoreip. The tool you use afterwards is fail2ban-client:

CONSOLE
$ sudo fail2ban-client status
$ sudo fail2ban-client status sshd             # currently banned IPs and counters
$ sudo fail2ban-client set sshd unbanip 203.0.113.9
flowchart LR
    L["auth.log entry:<br/>Failed password from IP"] --> R["Filter regex<br/>extracts the IP"]
    R --> C{"maxretry failures<br/>within findtime?"}
    C -- no --> W["Keep counting"]
    C -- yes --> B["Action: firewall rule<br/>drops traffic from IP"]
    B --> T["After bantime:<br/>rule removed"]

fail2ban turns log lines into temporary firewall rules.

The logic is easy to reproduce, and doing so shows how the window and the allowlist interact. This script reads a made-up log with documentation addresses (the ranges reserved for examples), applies maxretry = 3, findtime = 60 seconds and an allowlist, and prints each ban:

Bash A tiny fail2ban in Bash

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Notice that 198.51.100.4 is not banned: its two failures happened three minutes apart, outside the 60-second window, and a successful login followed. The address 192.0.2.10 failed three times and is untouched because it is on the allowlist. After a ban, the 10:04:10 attempt from 203.0.113.9 would normally never reach sshd at all; the line is printed to show the firewall dropping it. fail2ban cannot stop a distributed attack from thousands of addresses or protect a password that is correct, which is why keys, MFA and AllowUsers matter more.

Step 4: patch everything, automatically

Most breaches use vulnerabilities that already have a fix. Schedule security updates:

CONSOLE
# Debian / Ubuntu
$ sudo apt install unattended-upgrades
$ sudo dpkg-reconfigure --priority=low unattended-upgrades
# Fedora / RHEL family
$ sudo dnf install dnf-automatic
$ sudo systemctl enable --now dnf-automatic.timer

Kernel and glibc updates only take effect after a reboot (or live patching, offered by some vendors). needrestart (Debian family) and sudo dnf needs-restarting -r tell you when. Keep a test machine, apply updates there first for critical systems, and subscribe to your distribution's security announcements. Time must be correct too: enable chrony or systemd-timesyncd, because certificates and logs depend on it.

Step 5: permissions, setuid files and kernel settings

A setuid program runs with its owner's rights, usually root. passwd has to be one, since it writes /etc/shadow. But every setuid binary is an escalation path if it has a bug, so you should know exactly which ones exist. A baseline check is cheap: list them, compare with a known-good list, investigate differences. (Tools such as AIDE and Lynis automate this for the whole filesystem.) The sandbox is a real small Linux, so you can scan it:

Bash Find setuid files and compare with a baseline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The scan flags newgrp because we left it out of the baseline on purpose, to show what a difference looks like; on a real server you would check the package that owns it (dpkg -S, rpm -qf) and add it if it is expected. A real system will list different files, and that is the point of a baseline: an unexpected new setuid file, in /tmp or a home directory, is a classic sign of a compromise. Related checks: find / -xdev -type f -perm -0002 finds world-writable files, find / -xdev -nouser -o -nogroup finds files without an owner, and /etc/shadow should be 640 or tighter. Mount /tmp and /var/tmp with noexec,nosuid,nodev where practical.

The kernel itself has security-relevant tunables, such as disabling IP forwarding when you are not a router, ignoring ICMP redirects, enabling SYN cookies, and restricting the kernel log and pointers. They are set with sysctl, which the next lesson covers in Boot Process, GRUB, Kernel Modules and sysctl.

Step 6: audit and watch with auditd

If something changes, you want to know who changed it and when. The Linux Audit framework records security-relevant events in the kernel, and auditd writes them to /var/log/audit/audit.log. You declare what to watch in rule files under /etc/audit/rules.d/:

TEXT
# /etc/audit/rules.d/hardening.rules
-w /etc/passwd  -p wa -k identity
-w /etc/shadow  -p wa -k identity
-w /etc/sudoers -p wa -k sudo_changes
-w /etc/ssh/sshd_config -p wa -k sshd_config
-a always,exit -F arch=b64 -S execve -F euid=0 -k root_commands

A -w rule watches a path for the actions in -p (r read, w write, x execute, a attribute change), and -k gives the rule a searchable key. -a always,exit -S execve records every program execution (this can be very verbose). Load rules with sudo augenrules --load, list them with sudo auditctl -l, and query with:

CONSOLE
$ sudo ausearch -k identity -ts today         # events from the identity rules
$ sudo ausearch -m USER_LOGIN --success no    # failed logins
$ sudo aureport --auth --summary              # authentication report

Rules are easy to misread, so here is a small explainer that turns watch rules into sentences, exactly the way you would read them in a review:

Bash Explain audit watch rules

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Send audit logs and auth.log to a remote log host (rsyslog, journald remote, or a SIEM). Attackers who gain root delete local logs first, and logs that live elsewhere survive. Finally, run an automated audit such as sudo lynis audit system periodically; it grades your configuration and suggests concrete fixes. Keep tested backups, since hardening reduces risk but never to zero.

The short version

Area Do
SSH keys only, no root login, AllowUsers, sshd -t before reload
Firewall default deny incoming; open only what you serve
fail2ban bantime, findtime, maxretry in jail.local; allowlist your own IP
Accounts exact sudo rules, strong passwords or none, lockout, no shared accounts
Software automatic security updates; remove unused packages and services
Isolation SELinux or AppArmor enforcing; noexec,nosuid on /tmp
Visibility auditd watch rules, remote logs, periodic Lynis scan

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Your sshd_config contains PasswordAuthentication yes near the top and PasswordAuthentication no at the bottom. What does sshd use?

    In sshd_config the first obtained value for a keyword is used, so later lines are ignored. Verify the effective configuration with sshd -T.
  2. 2. What should you do right after editing sshd_config on a remote server?

    sshd -t checks syntax, and reload keeps existing sessions alive. Testing from a second terminal ensures you cannot be locked out by a bad change.
  3. 3. With maxretry = 3, findtime = 10m and bantime = 1h, when does fail2ban ban an address?

    findtime is the sliding window in which maxretry failures must happen, and bantime is how long the firewall rule stays in place.
  4. 4. Why is a new setuid file found under /tmp a serious finding?

    Legitimate setuid files are few and known. An unexpected one, especially in a writable directory, is a classic persistence or escalation technique.
  5. 5. What does the audit rule -w /etc/shadow -p wa -k identity do?

    -w sets a watch on a path, -p wa selects write and attribute-change events, and -k tags the events so you can find them with ausearch -k identity. Auditing records; it does not block.

Summary

  • Hardening is defense in depth: firewall, SSH, fail2ban, accounts, minimal services, MAC, updates and auditing, each limiting the damage when another layer fails.
  • SSH: keys only, PermitRootLogin no, AllowUsers, small MaxAuthTries; the first value of a keyword wins; check with sshd -t and sshd -T, and test from a second session.
  • Use a default-deny firewall, list listeners with ss -tlnp, and remove services and packages you do not need.
  • fail2ban turns log lines into temporary firewall bans (maxretry, findtime, bantime in jail.local); it complements, never replaces, strong authentication.
  • Automate security updates (unattended-upgrades, dnf-automatic) and reboot when kernel or core libraries change.
  • Keep a baseline of setuid files and sensitive file modes; unexpected changes deserve investigation.
  • auditd watch rules (-w PATH -p wa -k KEY) record who changed what; ship logs off the machine.

Next: Boot Process, GRUB, Kernel Modules and sysctl