Skip to content
Tutorials Center
My learning

Topics

Lesson 55 of 95 ยท Processes and the System

SSH, scp and Working Remotely

Use SSH safely: key pairs, ~/.ssh/config, scp and rsync, port forwarding, tmux, remote commands and scripts, and the quoting and stdin traps of automation.

Almost every Linux machine you will ever manage is somewhere else: a cloud server, a Raspberry Pi in a cupboard, a build box in another country. SSH (Secure Shell) is how you get a shell on it. It encrypts everything, proves who both sides are with cryptographic keys instead of passwords, and also carries file copies and network tunnels over the same connection.

Our sandbox has no network, no SSH server, and no user entry that ssh and ssh-keygen insist on, so you cannot make a real connection in the editor. This lesson is therefore half reference (read-only commands you will type on your own machine) and half runnable simulations: we build a public key by hand to see what it really is, resolve ~/.ssh/config rules in a script, reproduce the way SSH re-parses remote commands, copy a directory the way tar | ssh tar does, and run a script against several "hosts" to meet the classic stdin trap. Those parts are exactly how SSH behaves, and they are where scripts go wrong.

The basic commands

CONSOLE
$ ssh ada@server.example.com                 # interactive shell as user ada
$ ssh -p 2222 ada@server.example.com         # a non-standard port
$ ssh ada@server.example.com 'uptime'        # run ONE command and return
$ ssh -i ~/.ssh/id_work ada@server.example.com   # use a specific key

(All blocks in this section are read-only: they need a server.) The first form logs you in. The third form is the building block of remote scripting: run a command, get its output on your terminal (or in a pipe), and receive its exit status as the exit status of ssh itself. So ssh host 'test -f /etc/hosts' && echo "exists" works as you would hope.

If the remote command is missing or the connection fails, ssh itself exits with 255; any other value is the remote command's own status. Scripts that must tell the two apart can check for 255.

Keys instead of passwords

A password can be guessed and re-used; an SSH key pair cannot. You create two files:

  • the private key (~/.ssh/id_ed25519), which never leaves your machine, and
  • the public key (~/.ssh/id_ed25519.pub), a single line you may hand to anyone.

You append the public key to ~/.ssh/authorized_keys on the server. When you connect, the server sends a challenge that only the holder of the private key can answer. Nothing secret crosses the network.

CONSOLE
$ ssh-keygen -t ed25519 -C "ada@laptop"       # asks for a passphrase: use one
$ ssh-copy-id ada@server.example.com           # appends the public key to authorized_keys
$ ssh ada@server.example.com                   # no password needed any more

Choose ed25519 unless a very old system forces RSA (then use at least 3072 bits). Protect the private key with a passphrase and let ssh-agent hold the unlocked key in memory so you type the passphrase once per session (eval "$(ssh-agent)" and ssh-add; desktops start an agent for you).

ssh-keygen needs a user database entry for the current user, and the sandbox has none, so it fails here. We can still look inside a key, because a public key line is just text:

Bash What a public key really is

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

That SHA256:... string is what ssh-keygen -lf id_ed25519.pub and the first-connection prompt print. Comparing fingerprints is how you check that a key (or a server) is the one you expect without comparing 68 characters of base64. This key is a toy built from the bytes 1 to 32: never use made-up key material for real.

Permissions matter

OpenSSH refuses to use keys and config files that other users could read or change. The usual rules:

Path Mode Why
~/.ssh/ 700 only you may list it
private keys (id_ed25519) 600 readable only by you; ssh ignores the key otherwise
authorized_keys, config 600 the server rejects authorized_keys that others can write
public keys (*.pub) 644 public by nature

A tiny audit script, run on a fabricated .ssh folder, shows the idea (see File Permissions and Ownership):

Bash Auditing ~/.ssh permissions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The config file

Typing ssh -p 2222 -i ~/.ssh/id_web [email protected] gets old. ~/.ssh/config lets you give each machine a short alias and set defaults:

INI
# ~/.ssh/config   (read-only example)
Host web
    HostName 203.0.113.10
    User deploy
    Port 2222
    IdentityFile ~/.ssh/id_web

Host *.internal
    User ada
    ProxyJump bastion

Host *
    ServerAliveInterval 30
    IdentitiesOnly yes

Now ssh web does everything, and so do scp and rsync that mention web:. The matching rules are the part people get wrong, so we can model them. For each option, the first value found wins, and the file is read top to bottom over every Host block that matches, which is why specific blocks go first and Host * goes last.

Bash Resolving ssh_config rules: the first match wins

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Real ssh -G web prints the full effective configuration without connecting, which is the best debugging tool (here it is read-only, as ssh does not run in the sandbox). One more deliberate detail: web set Port 2222 and the later Host * block said Port 22, but the first one wins, so web uses 2222, while db.internal matches the *.internal block first and gets 2200.

Two settings deserve a place in nearly every config: ProxyJump bastion (connect through a jump host, replacing the older ProxyCommand incantations) and ControlMaster auto with ControlPath and ControlPersist 10m, which reuses one connection for many commands and makes scripts that call ssh in a loop dramatically faster.

Host keys: is this really the server?

On the first connection, SSH shows the server's host key fingerprint and asks whether to trust it; if you say yes it is stored in ~/.ssh/known_hosts. If the key later changes, SSH stops with WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!. That can be a reinstalled server, or a machine in the middle of your connection. Do not "fix" it blindly with StrictHostKeyChecking=no; verify the new fingerprint through another channel, then remove the old line with ssh-keygen -R hostname. In automation, pre-load the key (for example with ssh-keyscan plus a known-good fingerprint) instead of disabling the check.

Remote commands: the quoting trap

This is the most important piece of the lesson for script authors. When you run

CONSOLE
$ ssh web ls -l "my file.txt"

the local shell removes the quotes and hands ssh the words ls, -l, my file.txt. SSH then joins them with spaces into one string and sends that string to the server, where the user's login shell parses it a second time. The space in the name is lost on the way, and the remote ls sees two files. Everything you learned in Quoting and Expansion applies twice.

We can reproduce this exactly, with a function that does what the server does (join the arguments and parse again) on our own machine:

Bash Parsed twice: the remote shell re-splits your words

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Read the output carefully: the naive form fails because ls receives my and file.txt. The two fixes both make the string survive the second parse; printf '%q' is the robust one when the words come from variables. Also remember where a variable is expanded: in ssh web "echo $HOME" your local shell expands $HOME before SSH runs, while ssh web 'echo $HOME' (single quotes) sends the text and lets the remote shell expand it.

For anything longer than one command, send a script on stdin and let the remote Bash read it. Arguments go after --, and no second parse of the script text happens:

Bash Sending a script on stdin, the way ssh host bash -s does

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The same pattern lets you run a local function remotely: ssh web "$(declare -f myfunc); myfunc arg" sends the function's source followed by a call.

The stdin trap in loops

ssh reads standard input and forwards it to the remote command. Inside a while read loop that is a disaster: the first ssh call swallows the rest of the loop's input, so the loop runs for one host only, and nothing says why. The cure is ssh -n (or < /dev/null) for commands that need no input.

We can demonstrate it with a stand-in that, like ssh, consumes all of its stdin:

Bash Why the loop stops after the first host

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Running on many hosts, in parallel

Once the quoting and stdin are under control, fleet work is a loop with background jobs and wait, collecting each host's exit status (see Jobs and Background Processes). Each "host" below is just a directory so the example runs anywhere:

Bash One job per host, results collected afterwards

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Notice that each job writes to its own file, and results are printed in a fixed order afterwards. Interleaved output from parallel jobs is unreadable and a classic source of flaky scripts. For real fleets, tools like Ansible, pdsh or parallel-ssh do this for you with timeouts and inventories; shell loops are for a handful of machines.

Useful options in scripts: -o BatchMode=yes (never ask for a password or passphrase, fail instead), -o ConnectTimeout=5, -o ServerAliveInterval=15 (detect dead connections), -n (no stdin) and -T (no terminal). Use -t only when the remote command needs a terminal, such as sudo asking for a password or top.

Copying files: scp, rsync and tar over SSH

CONSOLE
$ scp report.pdf ada@web:/srv/reports/            # local to remote
$ scp -r ada@web:/var/log/app ./logs              # remote directory to local
$ scp -P 2222 file web:                           # NOTE: capital P for the port
$ rsync -avz --delete ./site/ web:/var/www/site/  # only changes, compressed
$ rsync -avzn ./site/ web:/var/www/site/          # -n = dry run: show, do nothing

scp is fine for a one-off. rsync is better for anything repeated: it transfers only the differences, can resume, and --dry-run previews a --delete. The trailing slash on the source decides whether the directory itself or only its contents are copied (site/ copies the contents; site creates site inside the target), the same trap as with cp in Copying, Moving and Deleting. Modern OpenSSH's scp uses the SFTP protocol internally, but its command line is unchanged.

When neither rsync nor scp suits, a tar pipe streams a directory tree through the connection, preserving permissions and symlinks, with no temporary archive:

CONSOLE
$ tar -C site -cf - . | ssh web 'tar -C /var/www/site -xf -'

The sandbox has no ssh, but tar piping works the same between two directories, and a checksum proves the copy:

Bash The tar pipe, with a local second host

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Port forwarding and tunnels

SSH can carry other TCP connections inside its encrypted channel. These are read-only: they need two machines.

CONSOLE
$ ssh -L 5433:localhost:5432 web      # local forward
$ ssh -R 8080:localhost:3000 web      # remote forward
$ ssh -D 1080 web                     # SOCKS proxy
TEXT
-L 5433:localhost:5432      "my port 5433 leads to port 5432 as seen FROM web"

  laptop                         web (server)
  psql -p 5433 ---+            +---> postgres :5432
                  |  encrypted |
                  +==== ssh ===+

-R 8080:localhost:3000      "web's port 8080 leads back to port 3000 on MY machine"

With -L, you reach a database that only listens on the server's loopback address, using a local client on port 5433. With -R, you expose a local development server through the remote machine. -D makes SSH a SOCKS proxy for a browser. Keep tunnels bound to loopback (the default) unless you have a strong reason; -N means "no remote command, just the tunnel", and -f sends it to the background.

Surviving disconnects: tmux and screen

An SSH session dies with the connection, and so does everything running in it. tmux (or the older screen) runs a terminal session on the server and lets you detach from it and reattach later. They are interactive, so the commands below are read-only:

CONSOLE
$ tmux new -s deploy       # start a named session on the server
  ...run the long job...   # press Ctrl-b then d to detach
$ tmux ls                  # list sessions
$ tmux attach -t deploy    # come back, even from another machine

For fire-and-forget jobs without a terminal, nohup command & or a systemd service is simpler; see Jobs and Background Processes.

How SSH secures the connection, and the server side

SSH combines three kinds of cryptography, and knowing which does what explains the warnings you see:

  1. Asymmetric encryption (key pairs: ed25519, RSA) proves identity. The server proves it owns its host key; you prove you own your private key without ever sending it. The same math is used once, at the start, to agree on a shared secret.
  2. Symmetric encryption (such as ChaCha20-Poly1305 or AES-GCM) protects all the traffic afterwards with that shared secret. It is much faster than asymmetric encryption, which is why it carries the bulk data.
  3. Hashing and message authentication detect tampering: every packet carries a checksum only the two ends can compute, so a changed byte is noticed and the connection is dropped.

To accept connections, a machine needs the server program sshd. On Debian and Ubuntu: sudo apt install openssh-server, then sudo systemctl enable --now ssh (the unit is sshd on Red Hat systems). Configuration lives in /etc/ssh/sshd_config: uncomment and set Port 22, PermitRootLogin no, PasswordAuthentication no (after keys work!) and AllowUsers alice. Check the file with sudo sshd -t and apply it with sudo systemctl reload ssh. Open the port in the firewall (sudo ufw allow OpenSSH or firewall-cmd --add-service=ssh --permanent) and test from a second terminal before closing your first one.

File transfer tools in more detail (all read-only here, they need two hosts):

CONSOLE
$ scp file.txt alice@host:/srv/data/               # local to remote
$ scp alice@host:/var/log/app.log .                # remote to local
$ scp -r -P 2222 project/ alice@host:/srv/         # directory, non-default port (capital -P)
$ scp host1:/tmp/a.txt host2:/tmp/                 # between two remote hosts
$ sftp alice@host                                  # interactive: ls, cd, get, put, mkdir, exit
sftp> put report.pdf
sftp> get -r results/
$ rsync -avz --ignore-existing src/ alice@host:/srv/dest/    # never overwrite what is already there
$ rsync -av --remove-source-files src/ alice@host:/srv/dest/ # a "move": delete sources after success

SFTP is a file-transfer protocol that runs inside SSH, so it needs no extra port and no separate server, and it supports resuming and directory listings, while plain old FTP sends passwords in the clear. scp is simple and fine for one-off copies; current OpenSSH versions implement it on top of the SFTP protocol. rsync sends only differences (its delta-transfer algorithm) and -a (archive) means -rlptgoD: recursive, links, permissions, times, group, owner and devices.

Security hygiene

  • Keys: ed25519, one per device, with a passphrase. Remove a lost device's public key from every authorized_keys.
  • Server (/etc/ssh/sshd_config, read-only here): PasswordAuthentication no, PermitRootLogin no (or prohibit-password), and optionally AllowUsers. Reload with systemctl reload ssh and keep a second session open while testing so a typo does not lock you out.
  • Agent forwarding (-A): a compromised server can then use your agent. Prefer ProxyJump for hopping through a bastion.
  • Never put private keys into Git repositories, container images or chat. Scan for them; revoke at once if one leaks.
  • Brute force: keep the SSH port reachable only from where you need it (firewall), or use a tool such as fail2ban.
  • Sudo and users are covered in Users, Groups and sudo.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which file must stay private and never be copied to a server?

    The private key proves your identity. The .pub file is meant to be shared and goes into authorized_keys on the servers you want to log in to.
  2. 2. You run ssh web ls -l "my file.txt" and ls complains about two missing files. Why?

    SSH sends one command string, and the remote login shell parses it. Quote it twice ("ls -l 'my file.txt'") or build it with printf '%q '.
  3. 3. A while read -r host; do ssh "$host" uptime; done < hosts.txt loop contacts only the first host. What is the fix?

    ssh forwards its standard input to the remote command, which eats the remaining lines of hosts.txt. -n connects stdin to /dev/null.
  4. 4. In an ssh config, Host web sets Port 2222 and a later Host * sets Port 22. Which port does ssh web use?

    ssh reads matching blocks from the top and keeps the first value for each option, so specific hosts go first and defaults under Host * go last.
  5. 5. What does ssh -L 5433:localhost:5432 web do?

    -L is a local forward: connections to your port 5433 travel through the encrypted channel and come out on web, which connects to localhost:5432 there. -R is the reverse direction.

Summary

  • SSH gives you an encrypted shell, file transfer and tunnels. Use key pairs (ed25519 with a passphrase and an agent); the private key never leaves your machine.
  • ~/.ssh and key files need strict permissions (700 for the directory, 600 for private keys and authorized_keys).
  • ~/.ssh/config defines aliases and defaults; the first value found for each option wins, so list specific hosts before Host *. ssh -G host shows the result.
  • Remote commands are parsed twice: quote for both shells, build strings with printf '%q', or send a script with ssh host 'bash -s -- args' < script.sh.
  • Use ssh -n in loops, -o BatchMode=yes and ConnectTimeout in scripts, and give each parallel job its own output file.
  • scp for one-offs, rsync (with --dry-run) for repeated syncs, tar | ssh tar for streaming trees; mind trailing slashes.
  • -L, -R and -D forward ports; tmux keeps sessions alive across disconnects; never disable host key checking just to silence a warning.

Next: The Order of Expansions in Depth