Working With Files in Bash: Globs, Safe Loops and Temp Files

Most shell scripts exist to do something to files: rotate logs, archive a directory, rename a batch of images, read a list of servers. The commands that copy, move and delete files are simple. The hard part is handing them the right file names, all of them, and nothing else, even when a name contains a space, starts with a dash or does not exist yet.

This part covers the Bash side of file handling: wildcard patterns and the options that change them, file names that break scripts, safe loops over files, file tests, reading and writing files, temporary files and Windows line endings.

Prerequisites

• The practice directory from Part 1 and the expansion order from Part 2.
• The basic file commands (cp, mv, rm, find, grep) from Linux command line basics. Permissions and links are in the file system part of the Linux guide.

Globbing: Wildcards in Depth

Globbing (pathname expansion) replaces a pattern with the names of matching files. Bash does it before the command runs, so ls *.txt gives ls a list of names, and ls never sees the *.

Pattern	Matches
*	Any string, including an empty one
?	Exactly one character
[abc]	One of a, b or c; [a-z] is a range
[!abc]	One character that is not a, b or c ([^abc] also works)
[[:digit:]]	One character of a class: alpha, digit, space, upper and so on

Three rules explain most glob surprises:

• Results are sorted by name, using the current locale's order.
• Hidden files are skipped unless the pattern itself starts with a dot. Since Bash 5.2, .* no longer matches . and .. (the globskipdots option).
• A pattern that matches nothing stays as it is. files/*.pdf with no PDF files is passed to the command as the literal text files/*.pdf.

The last rule bites in loops, which then run once with a name that does not exist. Two shell options change it:

Use nullglob in scripts that loop over files, and turn it off again afterwards if later commands rely on the default (ls *.pdf with nullglob on and no PDFs runs a bare ls, which lists everything).

More options, all set with shopt -s name:

Option	Effect
dotglob	* also matches hidden files
nocaseglob	Match without regard to case, so *.jpg finds photo.JPG
globstar	** matches any number of directories (Bash 4.0+)
extglob	Extended patterns: !(pat), @(a|b), *(pat), +(pat), ?(pat)

extglob has a catch: Bash must see shopt -s extglob before it parses the line that uses the pattern. On one line, it is too late:

In a script, put shopt -s extglob on its own line near the top, before any function that uses extended patterns.

Bash 5.3 adds GLOBSORT to sort the results by something other than the name, for example the newest file first:

The - reverses the order, so -mtime lists the most recently modified file first. On older Bash, sort with ls -t (only for display) or find -printf instead.

Braces Are Not Globs

Brace expansion looks similar but does something else: it generates words whether or not files exist. That makes it the right tool for creating names:

File Names That Break Scripts

Linux allows any character in a file name except / and the null byte: spaces, quotes, *, newlines and a leading - are all legal. Each one breaks a different kind of careless script.

Spaces split unquoted names into several arguments. A leading dash makes a name look like an option. Create a file called -l and watch ls * change its output format:

ls * expanded to ls -l data.csv monthly report.txt ..., and ls read -l as its long-listing option. A file called -rf in a directory where someone runs rm * does worse. Write ./* or put -- before file names whenever the names come from a glob or from user input; most commands, including rm, cp, mv, grep and tar, accept --.

Never loop over ls output. $(ls) is one string that Bash splits on whitespace, so a name with a space becomes two names:

Loop Over Files Safely

For files in one directory, a glob with nullglob is all you need. For a whole tree, use find and separate the names with null bytes, the one character that cannot appear in a name. There are three good ways to consume find output:

#!/usr/bin/env bash
# count-lines.sh: three safe ways to process every file that find returns

echo "1. find -exec runs the command with many names at once:"
find . -name '*.log' -exec wc -l {} +

echo "2. a while loop reading null-separated names:"
total=0
while IFS= read -r -d '' file; do
    lines=$(wc -l < "$file")
    total=$(( total + lines ))
    printf '   %-28s %3d\n' "$file" "$lines"
done < <(find . -type f \( -name '*.log' -o -name '*.txt' \) -print0 | sort -z)
echo "   total: $total lines"

echo "3. mapfile into an array (Bash 4.4+ for -d ''):"
mapfile -d '' -t files < <(find files -type f -print0 | sort -z)
echo "   ${#files[@]} files, the third is [${files[2]}]"

The pieces of the second method:

• -print0 makes find end each name with a null byte, and sort -z sorts null-separated input, so the order is stable.
• read -d '' reads up to the next null byte, -r keeps backslashes, and IFS= keeps leading and trailing spaces.
• < <(...) is process substitution: the loop reads find's output as if it were a file. Writing find ... | while read would also work, but the loop would run in a subshell and total would be 0 afterwards (Part 5 shows why).

find -exec ... {} + is the simplest choice when one command can take all the names. Use the loop when you need Bash logic for each file. Count lines of code recursively has more variations.

Test Files and Directories

[[ ... ]] has a test for every question you can ask about a path:

Test	True when
-e path	The path exists (any type)
-f path	It is a regular file (after following symlinks)
-d path	It is a directory (after following symlinks)
-L path	It is a symbolic link (even a broken one)
-s path	It exists and is not empty
-r, -w, -x	You can read, write or execute it
a -nt b	a was modified more recently than b (or b is missing)
a -ot b	a is older than b
a -ef b	Both names refer to the same file

Checking whether a directory is empty has no single test. Use a glob with nullglob and dotglob, so hidden files count too:

#!/usr/bin/env bash
# is-empty.sh: tell whether each directory given as an argument is empty
shopt -s nullglob dotglob

for dir in "$@"; do
    if [[ ! -d "$dir" ]]; then
        echo "$dir: not a directory"
        continue
    fi
    if [[ ! -r "$dir" || ! -x "$dir" ]]; then # an unlistable directory would look empty
        echo "$dir: permission denied"
        continue
    fi
    entries=("$dir"/*)
    if (( ${#entries[@]} == 0 )); then
        echo "$dir: empty"
    else
        echo "$dir: ${#entries[@]} entries"
    fi
done

Related posts: check if a file does not exist and get a file name's extension.

Read Files

The standard loop for reading a file line by line is while IFS= read -r line. Both parts of it matter:

Without IFS=, read strips leading and trailing whitespace. Without -r, it treats backslashes as escape characters and removes them. The || [[ -n "$h" ]] handles files whose last line has no newline, which many editors and Windows tools produce.

With a different IFS, read splits each line into fields. This reads the CSV inventory, skipping the header:

The braces share one redirection, so the first read takes the header and the loop gets the rest. This works for simple CSV files only; quoted fields with commas inside need a real parser (python3 -c 'import csv', mlr or csvkit).

Two more ways to read a file:

mapfile -t (also called readarray) reads lines into an array and -t strips the newlines. $(< file) is a faster $(cat file); like every command substitution, it removes trailing newlines.

Reading Configuration Files

A file of KEY=value lines can be loaded with source, but source runs the file as Bash code. Anyone who can write to the file can make your script run any command. For files you do not fully control, parse them instead:

#!/usr/bin/env bash
# read-config.sh: load KEY=value lines without executing the file
declare -A config=()

while IFS='=' read -r key value || [[ -n "$key" ]]; do
    [[ -z "$key" || "$key" == \#* ]] && continue      # skip blank lines and comments
    if [[ ! "$key" =~ ^[A-Z_][A-Z0-9_]*$ ]]; then
        echo "skipping invalid key: $key" >&2
        continue
    fi
    value="${value%\"}"; value="${value#\"}"             # remove surrounding double quotes
    config[$key]="$value"
done < "${1:-config/app.env}"

for key in APP_NAME APP_PORT LOG_LEVEL; do
    printf '%-10s = %s\n' "$key" "${config[$key]-<not set>}"
done

The second file tried to sneak a command into a key. Parsed, it is just a rejected line; sourced, it would have run. Part 4 explains the associative array and the ${value#\"} expansions.

Write Files

Part 2 covered > and >>. Three more techniques come up constantly:

A here document writes a block of text, with variables expanded unless you quote the delimiter:

tee writes to a file and to standard output at the same time; tee -a appends. It is also how you write to a root-owned file: sudo echo "x" > /etc/app.conf fails, because the redirection is done by your own shell before sudo starts, so use echo "x" | sudo tee /etc/app.conf > /dev/null.

Atomic replacement stops readers from ever seeing a half-written file. Write to a temporary file in the same directory, then rename it; mv within one file system replaces the target in a single step:

tmp=$(mktemp "config/.notes-api.XXXXXX") || exit 1
if generate_config > "$tmp"; then # any command that writes the new content
    chmod 644 "$tmp"               # mktemp creates files with mode 600
    mv -f -- "$tmp" config/notes-api.conf
else
    rm -f -- "$tmp"
    echo "generate_config failed; config/notes-api.conf is unchanged" >&2
    exit 1
fi

The if matters: if the generator fails halfway, the old file stays in place instead of being replaced by a broken one.

Temporary Files and Cleanup

Create temporary files with mktemp, never with fixed names such as /tmp/myscript.tmp or /tmp/out.$$. Predictable names let two runs overwrite each other and let other users on the machine plant a symlink in advance. mktemp creates a file with a random name and permissions 600 (mktemp -d creates a directory with 700):

#!/usr/bin/env bash
# tmpdemo.sh: a work directory that is removed however the script ends
workdir=$(mktemp -d "${TMPDIR:-/tmp}/tmpdemo.XXXXXX") || exit 1
trap 'rm -rf -- "$workdir"' EXIT

echo "working in $workdir"
grep ' 404 ' logs/access.log > "$workdir/404.txt"
echo "404 responses: $(wc -l < "$workdir/404.txt")"
stat -c '%A' "$workdir"            # the permissions: only you can enter it
exit 3                                  # even an error exit runs the trap

trap '...' EXIT runs its command whenever the script exits, whether it reaches the end, calls exit or stops on an error. Part 10 covers traps for signals and errors too.

Windows Line Endings

A file saved on Windows ends each line with a carriage return and a newline (CRLF). Bash treats the carriage return as part of the text, so comparisons fail for no visible reason, and a script with CRLF endings does not even start (Part 7 shows the errors):

${value@Q} prints the value quoted so hidden characters show; here it revealed the \r. sed -i 's/\r$//' or dos2unix fixes the file, and value=${value%$'\r'} fixes one variable.

Common Mistakes

• Looping over $(ls) or $(find ...). Names with spaces split. Use globs, or find -print0 with read -d ''.
• Forgetting that an unmatched glob stays literal. Use shopt -s nullglob or test [[ -e "$f" ]] in the loop.
• Passing glob results without -- or ./. A file called -rf becomes an option.
• while read line without IFS= and -r. Leading spaces and backslashes disappear.
• Sourcing a config file someone else can edit. source runs code; parse untrusted files.
• Fixed temporary file names. Use mktemp and a trap ... EXIT.
• sudo echo ... > /root-owned/file. The redirection runs as you; use sudo tee.

Summary

• Globs expand to sorted file names, skip hidden files, and stay literal when nothing matches; nullglob, failglob, dotglob, globstar and extglob change that.
• Braces generate names whether files exist or not.
• Protect file names with quotes and -- or ./, and never parse ls.
• For trees, use find -exec ... {} + or a while IFS= read -r -d '' loop over find -print0.
• Read lines with while IFS= read -r line, fields with IFS=,, and whole files with mapfile or $(< file).
• Use mktemp for temporary files, trap ... EXIT for cleanup and a rename for atomic writes.

Next: Bash Variables, with parameter expansion, arithmetic and arrays.

Keep Learning

• Loop through the lines of a file
• Check if a file does not exist
• List directories with their sizes
• How cat <<EOF works
• Interview questions: Linux file permissions