Bash Text Processing: printf, Here Docs, awk, sed and jq

Bash's strength is gluing programs together, and most scripts spend their time handing text from one tool to the next. Knowing which job belongs to a Bash builtin and which to awk, sed or jq is what keeps a script short, correct and fast.

This part covers printf and why it replaces echo in scripts, here documents, command and process substitution, a text-processing toolbox for logs (grep, cut, sort, uniq, awk, sed), how much faster builtins and awk are than loops, JSON with jq, parallel work with xargs and wait -n, curl in scripts, and dates.

Prerequisites

• Parts 2 to 8 of this guide, and the practice directory in ~/bash-practice.
• jq, curl and nc. On Debian and Ubuntu: sudo apt install jq curl netcat-openbsd; on macOS, jq comes from Homebrew and curl and nc are preinstalled.
• grep, find and pipes at the level of Linux command line basics. System tools such as ps, ss, tar and rsync are covered in the Linux guide and not repeated here.

printf Instead of echo

echo is fine for fixed text at the prompt. In scripts it has two problems: what it does with options and backslashes depends on the shell, and it cannot print a value that looks like an option.

Bash's echo treated -n as an option and printed nothing, and dash's echo interprets \t itself and prints -e literally. printf behaves the same everywhere: the first argument is a format, and the format is reused until all arguments are used up.

Rules of thumb: printf '%s\n' "$var" to print a variable, a format string in single quotes, and never a variable as the format (printf "$var" misbehaves when the value contains %). To write to standard error, add >&2 (echo to stderr).

Here Documents and Here Strings

A here document feeds the lines between <<WORD and a line containing only WORD to a command's standard input. How cat <<EOF works explains the mechanics; the variants that matter in scripts:

Form	Effect
<<EOF	Variables and $(...) are expanded; \ escapes $, ` and \
<<'EOF'	Nothing is expanded (quote any part of the word: 'EOF', "EOF", \EOF)
<<-EOF	Leading tab characters are removed, so the text can be indented
<<< "$text"	A here string: one string, plus a newline, as standard input

Here documents are the clearest way to send a block of commands or text to another program. For example, to run several commands on a remote server in one ssh connection, or to give psql a query:

# the quoted 'EOF' keeps $HOSTNAME for the remote shell to expand
ssh deploy@web-1.example.com bash -s <<'EOF'
set -e
cd /srv/notes-api
echo "deploying on $HOSTNAME"
EOF

psql "$DATABASE_URL" <<SQL
SELECT count(*) FROM orders WHERE created_at > now() - interval '1 day';
SQL

To put a multi-line text into a variable, use read -d '' or $(cat <<'EOF' ... EOF). Note that read -d '' returns status 1 when it reaches the end of the input, which stops a script under set -e unless you add || true:

Command Substitution

$(command) runs a command and becomes its output. The old form with backticks, `command`, still works but needs backslashes to nest and treats backslashes inside it differently, so use $( ):

Three details of command substitution cause surprises:

Null bytes cannot be stored in a variable at all; Bash drops them with a warning. Keep binary data in files.

Process Substitution

<(command) runs a command and gives you a file name (a pipe under /dev/fd) that reads its output. Use it wherever a program wants a file but you have a command:

You have already used the most common form, while read ... done < <(command), which keeps the loop in the current shell (Part 5). Process substitution is not part of POSIX: dash does not have it, and BusyBox ash has it only when built with Bash compatibility (Alpine's BusyBox 1.37 is). A >(...) command runs in the background, so zcat must not read requests.gz before gzip has finished writing it: $! holds the PID of the last process substitution, and wait $! waits for it.

A Text Toolbox for Logs

Most log questions are answered by a pipeline of small tools. With the access log fields (1 is the IP, 4 the time, 6 the method with a leading quote, 7 the path, 9 the status and 10 the size):

What each tool is for:

Tool	Use it to
grep	Keep or drop lines that match a pattern (-v, -c, -o, -E, -F)
cut	Take fields by a single-character delimiter
sort	Sort lines (-n numbers, -r reverse, -k by field, -u unique)
uniq -c	Count runs of identical lines (sort first)
awk	Work with fields, sums, counts and conditions in one pass
sed	Edit text in a stream: substitute, delete or print line ranges
tr	Replace or delete single characters
head, tail, wc	Take the first or last lines, count lines, words and bytes

Ubuntu and Debian install mawk as awk, Fedora and macOS use different implementations. Stick to POSIX awk features (no gensub, asort or strftime) when a script must run everywhere.

sed -i (edit a file in place) is the classic portability trap: GNU sed takes an optional suffix attached to -i, BSD sed on macOS requires one as a separate argument:

Builtins, Loops and awk: How Much Faster

Every $(...) that runs an external command starts a new process, and inside a loop that adds up. Here are two jobs done three ways, timed with Bash's time keyword on a small four-core ARM machine (your numbers will differ; the ratios are the point):

#!/usr/bin/env bash
# speed.sh: compare a builtin, an external command and awk
TIMEFORMAT='%R s'
path="/var/log/nginx/access.log.2.gz"

echo "10,000 times basename, with the \${path##*/} expansion:"
time for (( i = 0; i < 10000; i++ )); do f=${path##*/}; done
echo "10,000 times basename, with \$(basename):"
time for (( i = 0; i < 10000; i++ )); do f=$(basename "$path"); done

# a 20,000-line log: the practice log repeated 500 times
big=$(mktemp) || exit 1
trap 'rm -f -- "$big"' EXIT
for (( i = 0; i < 500; i++ )); do cat logs/access.log; done > "$big"
source lib/logstat.sh
echo "count by status, Bash loop with a regex per line (count_by):"
time count_by status "$big" > /dev/null
echo "count by status, awk:"
time awk '{ n[$9]++ } END { for (s in n) print n[s], s }' "$big" > /dev/null
echo "$f"

The parameter expansion took about a third of a second; starting basename ten thousand times took almost 40 seconds, roughly 4 ms per process on this machine (timings vary between runs, the ratios stay). awk counted the 20,000 lines in a few hundredths of a second, while the Bash loop in count_by, with a regex match and an array update per line, took about 18 seconds. The lesson is not "never loop in Bash": logstat is fast enough for log files of a few thousand lines, and it is easy to read and test. But when a script processes large files, let one awk, grep or sort process read the whole file instead of running commands per line.

JSON With jq

Do not parse JSON with grep, sed or cut: a reordered key, an escaped quote or a line break breaks them. jq parses it properly. Save a sample API response:

{
  "region": "eu-west",
  "services": [
    { "name": "notes-api", "port": 8080, "healthy": true, "replicas": 3 },
    { "name": "worker", "port": 9090, "healthy": false, "replicas": 2 },
    { "name": "cache", "port": 6379, "healthy": true, "replicas": 1 }
  ]
}

Build JSON with jq -n as well, so values are escaped correctly. A printf template breaks as soon as a value contains a quote:

The jq package in Ubuntu 24.04 is version 1.7.1, though it prints jq-1.7. The current release is jq 1.8.2 (June 2026), and Ubuntu 26.04 ships 1.8.1, which gives the same results for every filter in this part.

Run Work in Parallel

xargs -P runs a command on several inputs at once. With -0, it reads the null-separated names that find -print0 writes, so any file name is safe:

For Bash functions or more control, start jobs with & and limit how many run at once with wait -n (Bash 4.3+), which waits for any one job to finish:

#!/usr/bin/env bash
# parallel.sh: run 6 jobs, at most 3 at a time, and count the failures
max_jobs=3 running=0 failed=0

job() { # a stand-in for real work: sleep, fail on 5
    sleep 1
    (($1 != 5)) || return 1
    echo "job $1 done"
}

SECONDS=0
for n in 1 2 3 4 5 6; do
    if ((running >= max_jobs)); then
        wait -n || failed=$((failed + 1)) # collect one finished job
        running=$((running - 1))
    fi
    job "$n" &
    running=$((running + 1))
done
while ((running > 0)); do
    wait -n || failed=$((failed + 1))
    running=$((running - 1))
done
echo "finished in ${SECONDS}s, $failed job(s) failed"

Six one-second jobs finished in about two seconds instead of six. The script counts its jobs itself: each wait -n collects exactly one finished job and returns its exit status, including jobs that ended before wait -n was called, so every failure is counted. GNU parallel is a more complete tool for the same job if you can install it.

curl in Scripts

By default curl exits with 0 even when the server answers with an error such as 500, so a script cannot tell that the request failed. --fail (-f) turns HTTP errors into exit code 22. Here a one-line nc server returns a 500:

A good default for scripts is curl -fsS --retry 3 --max-time 10: fail on HTTP errors, no progress bar but still show errors, retry transient failures, and never hang. Newer curl versions (7.76+) also have --fail-with-body, which keeps the error response for logging.

Dates and Times

GNU date, on Linux, parses and formats dates; macOS uses BSD date, which has different options:

The sed only hides today's date. The equivalents on macOS:

Use UTC (-u, or TZ=UTC) for timestamps in logs and file names: they sort correctly and do not jump at daylight saving changes. See format a date as YYYY-MM-DD and the current time in seconds since the epoch.

Common Mistakes

• echo "$var" for arbitrary values. Use printf '%s\n' "$var".
• A variable as the printf format. Put it in an argument: printf '%s' "$var".
• Backticks, especially nested ones. Use $( ).
• Forgetting that $( ) strips trailing newlines and drops null bytes.
• Parsing JSON with grep or sed, or building it with printf. Use jq.
• A loop that starts a command for every line of a big file. Use one awk or grep.
• sed -i in a script that must run on macOS. Write a new file and mv it.
• curl without --fail in a script that checks whether a request worked.

Summary

• printf replaces echo in scripts; %q quotes, -v assigns.
• Here documents feed text blocks to commands; quote the delimiter to stop expansion, and remember read -d '' returns 1.
• $( ) captures output without trailing newlines; <( ) turns a command into a file name.
• cut, sort, uniq -c and awk answer most log questions in one line.
• Parameter expansion and one awk call beat per-line loops and external commands in loops by large factors.
• jq reads and writes JSON safely; xargs -P and wait -n run work in parallel; curl -fsS fails on HTTP errors.

Next: Bash Best Practices: strict mode and its traps, error handling, ShellCheck, tests with bats and portability.

Keep Learning

• How cat <<EOF works in Bash
• Set a variable to the output of a command
• Send output to a file and to the screen
• Sleep for milliseconds
• Interview questions: grep and text search