Bash Basics: How Bash Reads and Runs a Command Line

You type cp *.log "$backup_dir" and press Enter. Before cp starts, Bash has already split the line into words, replaced $backup_dir with its value, turned *.log into a list of file names and removed the quotes. cp never sees the * or the $. Knowing what Bash does in those steps, and in which order, is the difference between guessing why a command misbehaves and knowing.

This part follows a command line from the moment you press Enter to the moment the command exits: what kinds of commands exist, how exit codes work, the expansion order, quoting, lists and pipelines, subshells and redirection.

Prerequisites

• The practice directory from Part 1. Run the examples from ~/bash-practice.
• If cd, ls, cp and mv are new to you, read Linux command line basics first. This part is about Bash itself, not about individual commands.

The Shape of a Command Line

A simple command is a command name followed by words, which the command receives as arguments. Options are just arguments that start with - by convention; Bash does not treat them specially.

grep  -c  ' 404 '  logs/access.log  >  count.txt
│     │   │        │                └── redirection: handled by Bash, not passed to grep
│     │   │        └── argument 3
│     │   └── argument 2 (the quotes are removed before grep sees it)
│     └── argument 1 (an option)
└── command name

Bash also recognizes control operators that join commands: | (pipe), ; and newline (one after another), && and || (conditional), and & (run in the background). Everything else is built from these pieces.

Exit Status

Every command ends with an exit status, a number from 0 to 255. Zero means success, anything else means failure, and $? holds the status of the last command. if, while, && and || all decide based on it.

The conventions behind those numbers:

Exit status	Meaning
0	Success
1	General failure, or "false" (grep found nothing, a test failed)
2	Misuse: a bad option or argument (grep uses it for real errors too)
126	The file was found but could not be executed
127	Command not found
128 + N	Killed by signal N: 130 for Ctrl+C (SIGINT), 143 for SIGTERM

Your own scripts should follow them: exit 0 on success, 1 on failure, 2 for usage errors.

What Kind of Command Is It?

The first word of a command can be one of five kinds of thing, and Bash looks for them in this order: an alias, a keyword (if, [[, while), a function, a builtin (part of Bash itself, such as cd, echo, read) or a program file found in $PATH. type tells you which one you are getting:

Two practical consequences:

• echo is both a builtin and /usr/bin/echo, and they can behave differently. A bare echo runs the builtin, unless a function with the same name overrides it or the builtin was turned off with enable -n echo. The same is true for printf, test, kill, pwd and others, so read their documentation with help echo, not man echo, which describes the external program.
• To check whether a command exists in a script, use command -v name, not which. which is a separate program, its output format differs between systems, and it cannot see builtins, functions or aliases. See how to check if a program exists.

cd has to be a builtin: a separate program would change its own directory and exit, leaving your shell where it was.

The Order of Expansions

After splitting a line into words, Bash expands them in a fixed order. Each step works on the result of the previous one:

Step 3 covers parameter expansion ($name, ${name}), command substitution ($(...)) and arithmetic ($((...))), done left to right. Step 4 splits only the results of step 3 that are not inside double quotes. Step 5 expands every unquoted word that contains *, ? or [, whether you typed it or it came out of step 3. Together they are why quoting matters so much. Here is each step on its own:

set -x makes Bash print each command after expansion, prefixed with +, so you can see exactly what a program receives:

The order explains some surprises:

Because operators are recognized before expansion, a command stored in a string cannot contain |, >, && or quotes that work. Store commands in a function or an array instead (Part 4 and Part 8 show both). For ranges with a variable end, use seq or a C-style for (( i = 1; i <= n; i++ )) loop (Part 5).

Quoting

Quoting turns off special meanings. Bash has four kinds:

Form	What it does	Example
\x	The next character is literal	echo \$HOME
'single'	Everything is literal; you cannot put a ' inside	echo '$HOME and *'
"double"	$, ` and \ still work; no word splitting or globbing of the result	echo "$HOME and *"
$'ANSI-C'	Like single quotes, but \n, \t, \e and \' are interpreted	echo $'tab:\there'

The rule that prevents most bugs: put double quotes around every expansion ("$var", "$(cmd)", "${array[@]}") unless you want word splitting and globbing, and use single quotes for literal text. Quotes also join words: "monthly report.txt" is one argument, with the space inside it.

Lists, Pipelines and Grouping

A pipeline reports only its last command's status, so a failure on the left side goes unnoticed: above, grep failed with status 2, but the pipeline returned 0. ${PIPESTATUS[@]} holds every status of the last pipeline, and set -o pipefail makes a pipeline fail when any part fails (Part 10 covers it and its traps).

a && b || c is not an if-then-else. c also runs when b fails:

Use a real if when the middle command can fail.

Grouping runs several commands as one unit. Braces run them in the current shell; parentheses start a subshell, a copy of the shell whose changes vanish when it ends:

Note the syntax of braces: a space after { and a ; or newline before }. Subshells are also created for every part of a pipeline, for $(...) and for &. That is why a variable set inside cmd | while read ... is gone after the loop, a trap Part 5 shows how to avoid.

Redirection

Every process starts with three open file descriptors: 0 is standard input, 1 is standard output and 2 is standard error. Redirections, processed from left to right, point them somewhere else before the command runs:

Syntax	Effect
> file	Standard output to file, truncating it first
>> file	Standard output appended to file
< file	Standard input from file
2> file	Standard error to file
2>&1	Standard error to wherever standard output points right now
&> file	Both to file (Bash only; the same as > file 2>&1)
&>> file	Both appended to file
cmd1 |& cmd2	Both into the pipe (Bash only; the same as 2>&1 |)
> /dev/null	Discard the output
>| file	Overwrite even when set -o noclobber is on

The order of 2>&1 matters, because it copies where standard output points at that moment:

In the second command, 2>&1 pointed standard error at the terminal (where standard output still pointed), and only then did > out.txt move standard output. The error went to the screen.

To send everything a script prints to a log file from a given line on, redirect the shell itself with exec:

#!/usr/bin/env bash
exec >> /var/tmp/nightly.log 2>&1   # every later command writes here
echo "started at $(date -u +%FT%TZ)"

Redirect all output to a file, send output to a file and the screen with tee and what 2>&1 means go further. Here documents (<<EOF) and here strings (<<<), which feed text to standard input, are in Part 9.

History Expansion and Line Editing

In an interactive shell, Bash keeps a history and lets you reuse it. These are the shortcuts worth learning first:

Type this	Bash does
Ctrl+R	Search backwards through history; press again for older matches
!!	The previous command, for example sudo !!
!$	The last argument of the previous command (or press Alt+.)
^old^new	Rerun the previous command with old replaced by new
Ctrl+A / Ctrl+E	Jump to the start / end of the line
Ctrl+W	Delete the word before the cursor
fc	Open the previous command in your editor, run it when you save

Two settings in ~/.bashrc make history more useful: HISTCONTROL=ignoreboth skips duplicates and commands that start with a space (handy for commands that contain a secret), and HISTTIMEFORMAT='%F %T ' adds a timestamp to history output. History expansion with ! only happens in interactive shells, so ! in a script is an ordinary character. If it gets in your way interactively, for example in echo "Hello!", use single quotes or turn it off with set +H.

Getting Help

• help lists the builtins; help read shows the options of one. Bash builtins are not documented in their own man pages.
• man bash is the complete reference, more than 6,000 lines long. Search it with / followed by a word, for example /Parameter Expansion.
• type name shows what a name is; man name is for program files.

Common Mistakes

• Expecting a variable to set a brace range ({1..$n}). Braces expand first; use seq or a C-style loop.
• Storing a command with pipes or quotes in a string and running $cmd. Use a function or an array.
• Unquoted expansions. $var and $(cmd) without double quotes are split on whitespace and globbed.
• Writing 2>&1 > file when you meant > file 2>&1.
• Using a && b || c as if/else. c runs when b fails.
• Reading man echo for the builtin. Use help echo.

Summary

• A command line is split into words and operators, then expanded in order: braces, tilde, $ expansions, word splitting, globbing, quote removal.
• Word splitting applies to unquoted expansion results, and globbing to every unquoted pattern, including ones that came from a variable. Double-quote your expansions.
• type shows whether a name is an alias, keyword, function, builtin or file; use command -v in scripts.
• Exit status 0 is success. Pipelines report the last command's status unless pipefail is on.
• Braces group commands in the current shell, parentheses in a subshell whose changes are lost.
• Redirections are processed left to right, so > file 2>&1 sends both streams to the file.

Next: Working with Files, where globbing, safe file names and temporary files get the attention they need.

Keep Learning

• What does 2>&1 mean?
• Redirect all output of a script to a file
• Echo to standard error
• Linux command line basics in the Linux guide