Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 95 ยท Pipes and Redirection

Here-Documents, Here-Strings and Input

Learn Bash here-documents (<<EOF, <<-, quoted delimiters) and here-strings (<<<), how to read stdin with read and cat, and how to generate files and templates.

So far, standard input came from a file (< file) or from another command (|). But very often a script needs to feed a block of text to a command that is written right inside the script: the body of a config file, the lines of a test input, an SQL script, a usage message. Writing that text with a dozen echo commands is clumsy, especially when it contains quotes. Bash has two features for it. A here-document (<<EOF) turns the following lines of the script into the command's stdin. A here-string (<<<) does the same for a single string.

This lesson also covers the other half of the topic: how a script reads its standard input with read and cat, how to use the Input box of our editor, and how to build files from templates.

Here-documents: the basics

A here-document starts with << followed by a delimiter word. Everything on the following lines, up to a line that contains only that word, becomes the standard input of the command:

TEXT
command <<DELIMITER
...lines of text...
DELIMITER

The delimiter is just a word you choose; EOF ("end of file") is the habit, but END, TEXT or SQL are fine, and a descriptive one documents the content. The closing delimiter must be on a line by itself, starting in the first column (no leading spaces, no trailing spaces) unless you use <<-.

Bash A first here-document

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

cat > file <<EOF is the classic way to write a file from a script. The redirections > and << can appear in any order. The here-document body is stored by the shell and handed to the command through a pipe or a temporary file; to the program it is indistinguishable from ordinary input.

Expansion inside a here-document

By default the body is treated like text in double quotes: $variable, ${...}, $(command), $((arithmetic)) and backslash escapes are expanded before the command sees the text. That is exactly what you want for templates.

Bash Variables and commands expand in an unquoted here-document

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Quotes are not special in the body, so you can write "double" and 'single' quotes freely, which is the main reason to prefer a here-document over echo.

Quoting the delimiter turns expansion off

If you quote any part of the opening delimiter (<<'EOF', <<"EOF" or <<\EOF), the body is taken literally: no variable, command or arithmetic expansion, and backslashes stay. Use this when the text contains $ characters that must reach the program unchanged, for example when you write a shell script, a Makefile, a Dockerfile or an awk program with a here-document.

Bash <<EOF versus <<'EOF'

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

When you generate a script this way, the quoted delimiter keeps $@, $who and friends intact in the file. With an unquoted delimiter the shell would substitute them (with empty values) while writing the file, and you would get a broken script.

Indentation with <<-

Scripts indent their code, and a here-document inside an indented block looks ugly if its text and delimiter must start in column 1. The form <<- strips leading tab characters (only tabs, never spaces) from the body lines and from the delimiter line. Then you can indent the whole block to match your code.

Bash <<- strips leading tabs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The source of this example is indented with real tab characters. If your editor converts tabs to spaces, <<- stops working, the delimiter is not recognized and the shell complains that the here-document "delimited by end-of-file" was never closed. Because of that fragility, many projects avoid <<-, or keep that part of the file tab-indented via .editorconfig.

Here-strings: <<<

command <<< word feeds a single string, followed by a newline, to the command's stdin. The word undergoes the usual expansions (variables, command substitution) but no word splitting. It is the convenient replacement for echo "$var" | command, and it avoids creating a pipeline subshell.

Bash Here-strings

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The two lines about the variables show an important gotcha (after a pipe: p='' q='' r=''): echo "$text" | read ... runs read in a subshell (every pipeline stage is a separate process), so the variables are lost when the pipeline ends. read ... <<< "$text" runs in the current shell. This is explained further in Subshells, Groups and Environment. Also note that <<< always appends a newline; od -c shows a b \n.

Here-strings are a Bash/zsh/ksh feature, not part of POSIX sh. In portable scripts use a here-document or printf '%s\n' "$text" | command.

Where here-documents go in a command line

A here-document attaches to one command, but the command can be part of a pipeline, a loop or a function, and you can keep writing the rest of the command line on the line with <<. The body starts on the next line.

Bash Here-documents inside pipes and constructs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Notice how done <<EOF feeds the whole loop, just like done < file. Two here-documents on one line are rare and confusing; prefer separate commands.

Reading standard input in scripts

The other side: a script that consumes what is fed to it. There are three main tools.

  • cat reads all of stdin as a block: data=$(cat).
  • read takes one line (more precisely, up to a newline) into one or more variables. Always use read -r so that backslashes are kept as they are, and IFS= read -r line to preserve leading and trailing blanks. read returns a non-zero status at end of file.
  • Any filter (sort, awk, wc) simply reads stdin when it has no file argument.

The Input box of our editor supplies stdin: whatever you put there is what the script reads. In a lesson you can preset it, as here.

Bash Reading stdin with read, cat and while

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Each read consumes one line, and later commands continue from where the previous one stopped, because the three commands share one open stdin. Run it and you see the first two lines taken by read and the third by cat. Reading in a loop until end of file is the idiom while IFS= read -r line; do ...; done.

Bash Reading until end of file and splitting fields

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

IFS=, set just for the read command makes comma the field separator, so a CSV line is split into three variables; the last variable receives whatever remains. See Reading Input and Interactive Scripts for prompts (-p), silent input (-s), timeouts (-t) and detecting a terminal with [[ -t 0 ]].

Do not run read on the output of a pipe when you need the variable later: cmd | read x loses x. Use a here-string (read x <<< "$(cmd)"), process substitution (read x < <(cmd)) or shopt -s lastpipe.

Combining here-documents and stdin

If a command reads from a here-document, its stdin is used up; to give a script both a data block and the real input, use another file descriptor (as in Standard Streams and Redirection) or cat to join sources. A neat trick is cat file - <<EOF, where - stands for stdin.

Bash Mixing a here-document with piped data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Generating files and templates

Here-documents shine for generating configuration files, because variables fill in the blanks. The pattern: set variables, write the file with an unquoted delimiter. Escape the characters that must stay literal (\$).

Bash A configuration generator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

For large templates, tools like envsubst (from the gettext package, not installed everywhere) replace ${VARIABLE} markers in a template file. A tiny Bash version is possible with eval, but beware: eval on text you did not write is a security hole, see The Order of Expansions in Depth.

SQL, messages and other embedded text

Here-documents also let a script talk to other programs in their own language without escaping trouble. The sandbox has no database client, but the idea is the same for sqlite3, mysql or psql:

CONSOLE
$ sqlite3 shop.db <<'SQL'
CREATE TABLE IF NOT EXISTS items (id INTEGER PRIMARY KEY, name TEXT);
INSERT INTO items (name) VALUES ('pen');
SELECT COUNT(*) FROM items;
SQL

The block above is read-only (no sqlite3 binary in our runner). The quoted delimiter protects the SQL from the shell. For text messages, a usage function is the standard case:

Bash A usage message with a here-document

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

${0##*/} strips the directory from the script name (see Parameter Expansion and String Handling). A usage function that prints to stdout for --help and to stderr (usage >&2) on a mistake is a convention worth copying.

Pitfalls checklist

  • Trailing whitespace after the closing delimiter (or spaces before it) means the delimiter is not recognized and the shell reads to the end of the file: "here-document delimited by end-of-file".
  • Unquoted delimiter + $ in the body: the shell expands. Quote the delimiter to be literal.
  • <<- needs tabs. Spaces do not work.
  • Pipelines and variables. cat <<EOF | read x loses the variable; use a here-string or process substitution.
  • Here-documents and set -e/-u. An unset variable inside an unquoted body fails under set -u; use ${var:-} where empty is allowed.
  • Windows line endings. If the file has \r\n, the delimiter line is EOF\r and never matches. Convert with tr -d '\r' or dos2unix.
  • Large inputs. Here-documents are for small blocks. For big data, keep them in files.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the difference between cat <<EOF and cat <<'EOF'?

    An unquoted delimiter allows $variable, $(command) and $((math)) expansion in the body. Quoting any part of the delimiter turns expansion off.
  2. 2. What does <<-EOF do to the body of a here-document?

    The dash strips leading tabs (only tabs) so that the block can be indented with the surrounding code.
  3. 3. Why does echo "a b" | read x y; echo "$x" print an empty line in Bash?

    Each stage of a pipeline is a separate process. Use read x y <<< "a b" so that read runs in the current shell.
  4. 4. Which command reads one line of stdin into line and keeps backslashes and surrounding blanks as they are?

    -r disables backslash escapes and the empty IFS stops read from trimming leading and trailing whitespace.
  5. 5. A script gives the error here-document delimited by end-of-file. Which is a likely cause?

    The closing delimiter must be alone on its line. Extra spaces, a carriage return or tab-versus-space mistakes with <<- mean the shell never finds it.

Summary

  • <<DELIM makes the following lines, up to a line containing only DELIM, the command's stdin; cat > file <<EOF writes a file from a script.
  • An unquoted delimiter expands $var, $(cmd) and $((expr)); quote it (<<'EOF') for literal text such as generated scripts.
  • <<- strips leading tabs so a block can be indented; it never strips spaces.
  • <<< "text" is a here-string: one string plus a newline as stdin, in the current shell, so read a b <<< "$text" keeps its variables.
  • Read input with IFS= read -r line, while IFS= read -r line; do ... done, or cat; the Input box of the editor feeds stdin.
  • Use here-documents for config templates, usage messages and embedded SQL; keep the closing delimiter alone on its line.

Next: xargs, Command Substitution and Process Substitution