Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 95 ยท Working with Text

sed: The Stream Editor

Learn sed on Linux: s/// substitution and flags, addresses and ranges, d and p, -n, safe in-place editing with -i, multiple commands, hold space and pitfalls.

sed is the stream editor. It reads text one line at a time, applies the editing commands you give it, and writes the result to standard output. It does not open an editor window or ask questions: you describe the change ("replace every colour with color", "delete lines 1 to 3", "print only the lines between two markers") and sed applies it to a file or to a pipeline. That makes it perfect for scripts, quick fixes and bulk edits.

Most people use sed for one command only, s/old/new/, and that command alone is worth the lesson. But sed can also select lines by number or pattern, delete, insert and rearrange text, and even work across lines. You will need regular expressions here, so keep that lesson at hand. All examples use GNU sed 4.7; the differences from macOS/BSD sed are listed at the end.

How sed works

sed repeats the same cycle for every input line:

TEXT
read a line into the "pattern space"
   -> run every command on it, in order
   -> print the pattern space (unless -n was given)
   -> next line

The original file is never touched unless you ask with -i. By default, sed prints the edited text, so you can look at the result first. The general shape is:

TEXT
sed [options] 'address command' [file...]

Commands are mostly one letter: s substitute, d delete, p print, a/i/c append/insert/change, y transliterate, q quit. An address in front of a command (a line number, a pattern or a range) limits which lines it affects; without one the command applies to every line.

s: substitution

The substitute command is s/pattern/replacement/flags. It replaces the first match of pattern on each line.

Bash The s command and its flags

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

What to remember about s:

  • Flags: g all matches on the line, a number (2) the nth match, I case-insensitive, p print the line if a substitution happened, w file write such lines to a file. Flags combine: s/a/b/2g replaces the second match and every later one.
  • Delimiter: the character after s can be any character, so s|/usr|/opt| is easier than escaping slashes (s/\/usr/\/opt/). Common choices are |, # and ,.
  • BRE by default: +, ?, | and () are literal unless you add -E (or escape them). I use -E in nearly every example.
  • & in the replacement stands for the whole match, \1..\9 for the groups, and \n for a newline (GNU). A literal & or \ in the replacement needs a backslash.
  • Case conversion (GNU): \U, \L change the following text, \E ends it: sed -E 's/\b(.)/\U\1/g' capitalizes every word.

The pattern and replacement are not plain strings. If you build them from a variable (sed "s/$old/$new/") and the variable contains /, &, \ or regex characters, the command breaks or does something unexpected. Quote carefully, choose a delimiter that cannot occur in the data, and for a plain-text replacement consider awk or a small while read loop instead.

Addresses: choosing lines

An address before a command selects the lines it acts on.

Address Selects
5 line 5
$ the last line
2,4 lines 2 through 4
3,$ line 3 to the end
/regex/ every line matching the regex
/start/,/end/ from a line matching start to the next matching end, repeatedly
0~2 or 1~2 every second line (GNU: first~step), 0~2 is even lines
2,+3 line 2 and the next 3 lines (GNU)
addr! every line NOT selected
Bash Addresses and ranges

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The last command uses an empty pattern //, which means "the last pattern used", to skip the lines that matched BEGIN or END. Curly braces group several commands under one address, as in /BEGIN/,/END/{ ... }.

d, p and -n: deleting and printing

d deletes the line (it is not printed, and sed starts the next cycle). p prints the line an extra time, so with the normal automatic print each selected line appears twice. That is why p is almost always combined with -n, which turns the automatic print off, so only what you explicitly p appears.

  • sed '3d' removes line 3; sed '/^#/d' removes comments; sed '/^$/d' removes blank lines.
  • sed -n '3p' prints only line 3; sed -n '5,10p' a range; sed -n '$p' the last line.
  • sed -n '/error/p' behaves like grep error.
  • sed '1d' skips a header; sed '$d' drops the last line.
  • sed 5q prints five lines and quits, a fast head -n 5; it also stops reading, which matters on huge files.
Bash d, p, -n and q

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

sed -n '$=' is a cute way to count lines (= prints the current line number, and $ selects the last line). n replaces the pattern space with the next input line, so {n;p} prints the line after a match.

a, i, c, y: adding and changing whole lines

  • a text appends a line after, i text inserts before, c text replaces the selected line(s).
  • y/abc/xyz/ transliterates characters one for one (like tr).
  • One-liners with these commands are a GNU extension. POSIX requires the text on the next line after a backslash; the GNU form sed '2a new line' is fine on Linux.
Bash Insert, append, change and transliterate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Editing files in place with -i

So far sed only wrote to the screen. The -i option ("in place") rewrites the file itself. It is powerful and dangerous, so learn the safe pattern: give it a backup suffix.

Terminal
sed -i.bak 's/old/new/g' file       # edits file, keeps the original as file.bak

Without the suffix (sed -i ...) the original is gone. Internally sed writes a temporary file and renames it over the original, so the file's inode changes, symbolic links are replaced by regular files unless you pass --follow-symlinks, and a hard link no longer shares content.

Bash Safe in-place editing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Good habits for in-place edits:

  1. Run the command without -i first and check the output.
  2. Use -i.bak (or work in a Git repository, where git diff is your safety net; see the Git tutorial).
  3. Do not read and write the same file with redirection: sed 's/a/b/' f > f truncates f before sed reads it and destroys the file. Use -i or write to a temporary file and mv it.
  4. macOS/BSD sed -i requires an argument (sed -i '' 's/a/b/' f), so scripts that must run on both systems should write to a temp file instead.

Several commands, scripts and files

You can chain commands in four ways:

Terminal
sed 's/a/b/; s/c/d/' file                 # separated by semicolons
sed -e 's/a/b/' -e 's/c/d/' file           # separate -e options
sed -f script.sed file                     # commands in a file
sed -n '/x/{p;q}' file                     # braces group commands for one address

Commands run in order on each line, so the second sees what the first produced. A sed script file (-f) is easier to read for anything longer than a line, and it may contain comments with #. Its first line can even be #!/bin/sed -f to make it executable.

Bash A sed script file

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The last pipeline uses two more tricks: = prints each line number on its own line, and N joins the next line onto the current one so s/\n/: / can replace the newline between them. N leads straight to sed's advanced feature.

Hold space: remembering text across lines

Besides the pattern space (the current line), sed has a second buffer, the hold space. You copy and swap between them with these commands:

Command Action
h / H copy / append the pattern space to the hold space
g / G copy / append the hold space to the pattern space
x exchange both
N append the next input line to the pattern space (with a newline)
D / P delete / print up to the first newline

Their classic use is rearranging lines. The following reverses a file like tac, one of the most famous sed idioms: 1!G;h;$!d. Read it as "from line 2 on, append the hold space (the previous lines in reverse) to the current line; always store the result in the hold space; do not print anything except at the end".

Bash Hold space recipes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

:a;N;$!ba;s/\n/,/g defines a label a, appends the next line, and jumps back (b a) until the last line ($! means "not the last"), then replaces all newlines. Hold-space programs are compact but hard to read; if a sed command needs more than a line of explanation, write it as an awk program or a small script instead (see awk). For the last example we deliberately switched to Bash because sed is poor at arithmetic.

Pitfalls and when to avoid sed

  • Quoting. Use single quotes around the script. In double quotes the shell expands $, backticks and \ first. Use double quotes only to insert a variable, and escape what must stay literal.
  • Greedy matching. sed 's/".*"/X/' replaces from the first quote to the last. Use "[^"]*" (see Regular Expressions).
  • sed -i in a pipeline or on special files replaces them with regular files. Use --follow-symlinks for links.
  • Multi-line edits are awkward. Matching across line boundaries needs N loops or -z (GNU: treat the whole file, NUL-separated, as one line): sed -z 's/a\nb/X/' file.
  • Special characters in replacement. & and \ are special; a newline in the replacement must be written \n (GNU) or as a backslash-newline pair.
  • sed -n without p prints nothing and looks like a bug.
  • BSD/macOS sed lacks I, \+, \?, \|, 0~2, -z, \U, one-line a/i/c, and requires -i ''. Prefer GNU sed via gsed, or restrict yourself to POSIX features (-E is available in macOS sed).
  • Do not parse structured data with sed. For JSON use jq; for HTML, a parser; for CSV with quoted fields, a CSV tool.

Choose sed for line-oriented substitution and deletion, grep to only select lines, and awk when you need to work with columns or numbers.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does sed 's/a/b/' file do to the file?

    Without -i, sed only writes to standard output. Without the g flag it replaces just the first match on each line.
  2. 2. Which command prints only lines 3 to 5?

    -n turns off the automatic printing and p prints the selected lines. Without -n, lines 3-5 would be printed twice and all other lines once.
  3. 3. You run sed 's/old/new/' data.txt > data.txt. What happens?

    The shell opens and truncates the output file before starting sed, so sed reads an empty file. Use sed -i, or write to a temporary file and rename it.
  4. 4. How do you replace /usr/local with /opt without escaping slashes?

    The character after s is the delimiter; any character works, so | avoids the leading-slash escapes.
  5. 5. What does the sed script 1!G;h;$!d do?

    For lines after the first it appends the hold space (the earlier lines in reverse), stores the result in the hold space, and prints only at the end, reproducing tac.

Summary

  • sed reads lines, applies commands and prints the result; the file is only changed with -i.
  • s/pattern/replacement/flags is the core command: flags g, I, p, a number; & and \1 in the replacement; any delimiter; -E for extended regex.
  • Addresses select lines: numbers, $, /regex/, ranges a,b, first~step, and ! to negate.
  • d, p, -n, q, a, i, c, y delete, print, stop, add and change lines.
  • Use -i.bak, test without -i first, and never redirect output into the input file.
  • Combine commands with ;, -e, braces or -f script.sed; the hold space and N handle multi-line edits.
  • GNU and BSD sed differ; prefer awk or a real parser for structured data.

Next: awk: Column Processing and Reports