Advanced Git: Worktrees, Sparse Clones, Submodules

You can use Git for years with the commands from the earlier parts. This part is for the moments when they are not enough: a repository too big to clone comfortably, two branches you need open at the same time, the same conflict for the fifth time, another project's code inside yours, or a script that has to parse Git's output. It starts with how Git stores data, because the rest builds on it, and covers the SHA-256 and reftable formats that become defaults in Git 3.0.

Prerequisites

• The status-page repository from Part 9, with its remote in ~/server.
• Comfort with branches, merging, rebasing and remotes (Parts 4 to 6).

How Git Stores Data

Everything in .git/objects is one of four object types, each named by the hash of its content:

• A blob is the content of one file, without its name.
• A tree lists names, modes and the hashes of blobs and other trees: one directory.
• A commit points to the top-level tree, its parent commits, and records the author, committer and message.
• An annotated tag points to a commit and records the tagger and message (and a signature).

git cat-file reads any object:

An object's name is the hash of a short header plus its content, which you can reproduce:

(On macOS, use shasum -a 1 instead of sha1sum.)

Because a commit contains its tree's hash and its parents' hashes, changing anything in the past (one byte of one file) changes every hash after it. That is why rewriting history gives commits new hashes, and why a signed commit or tag protects everything it points to.

Refs are names for hashes: branches in refs/heads/, remote-tracking branches in refs/remotes/, tags in refs/tags/. Objects start as individual compressed files ("loose objects") and are packed into pack files with delta compression by git maintenance (Part 9):

To find what makes a repository big, list the largest blobs in the whole history, with their size in bytes and path (files that were deleted long ago count too):

SHA-256 and Reftable Repositories

Git has supported repositories that name objects with SHA-256 since Git 2.29, and dropped the "experimental" label in 2.42. SHA-1 is still the default; Git uses a hardened SHA-1 that detects the known collision attacks, but SHA-256 is the long-term answer, and Git 3.0 makes it the default for new repositories (Part 1). Create one with --object-format:

The two formats cannot exchange commits yet: Git's work on converting between them is still in progress, so a SHA-256 repository can only push to and fetch from other SHA-256 repositories. Hosting support is uneven. Forgejo and Codeberg support SHA-256 repositories, GitLab has offered them as an experiment since 2024 (GitLab's announcement), and GitHub does not offer them as of October 2026 (the open community discussion). For a project that lives on GitHub, stay with SHA-1 for now.

Reftable is a new storage format for refs: a short stack of binary tables (listed in tables.list) instead of one file per ref plus packed-refs. It is faster with many refs, atomic when several refs change at once, and handles branch names that differ only in case on Windows and macOS. Git 3.0 makes it the default too. You can create a repository with --ref-format=reftable, or convert one:

Everything works as before; only tools that read .git/refs files directly (instead of running git rev-parse or git for-each-ref) notice the difference.

Worktrees: Several Branches Checked Out at Once

Switching branches changes the files in your one working tree. A worktree is a second (or third) working tree attached to the same repository, with a different branch checked out. You can review a colleague's branch, run a long test suite on one branch while you edit another, or let several coding agents work on separate branches of one clone, without stashing or a second clone:

All worktrees share the objects, branches, stashes and configuration (the services.txt hook ran in the second worktree too), so a commit in one is visible in the others at once. Each has its own HEAD, index and files. git worktree remove deletes the folder; if you delete it by hand, git worktree prune cleans up the record.

Work with Large Repositories

When a repository has years of history or thousands of files, three features cut what you download and check out. Hosting services enable the server side for you; for a plain bare repository, allow filters once:

Partial clone and sparse checkout

A partial clone with --filter=blob:none downloads all commits and trees but no file contents; Git fetches each file's content the first time it needs it. Sparse checkout limits the working tree to the directories you name. Together they give you a small, fast clone of a big repository:

--sparse starts with only the files at the top level (in cone mode, the default, you list directories). git sparse-checkout set docs added the docs directory and fetched just the blobs it needed. History commands still see every commit; only file contents arrive lazily. git sparse-checkout disable brings back the full working tree, and Git 2.52's git sparse-checkout clean removes files left behind outside the chosen directories (preview with --dry-run, then delete with -f).

Clone	Downloads	Good for
--filter=blob:none	All commits and trees; file contents on demand	Daily work in big repositories
--filter=tree:0	All commits; trees and contents on demand	CI jobs that build one commit and need git log
--depth=1	One commit with its files	CI jobs that only build; no history
--sparse (with a filter)	Only the directories you choose	Monorepos where you work in a few directories

Partial clones need a connection to the remote whenever Git fetches missing content, for example for git log -p over old commits. git backfill (Git 2.49+, experimental) downloads missing blobs in batches: by default those reachable from HEAD, and in a sparse checkout only the ones inside your directories (--no-sparse lifts that; since Git 2.54 it also takes a revision range).

For very large repositories, the scalar command that ships with Git sets all of this up in one step (scalar clone <url>), including background maintenance and a file system monitor. The built-in fsmonitor (git config core.fsmonitor true) makes git status fast in big working trees by asking the operating system which files changed instead of scanning them all; it has existed on macOS and Windows for years and came to Linux in Git 2.55.

Reuse Conflict Resolutions with rerere

rerere ("reuse recorded resolution") remembers how you resolved a conflict and applies the same resolution when the same conflict comes back: when you rebase a long-running branch again, redo a merge, or test-merge branches before the real merge. It is off by default; Part 9 turned it on with rerere.enabled true. Set up a conflict in the page title:

"Recorded preimage" means rerere saved the conflict. When you committed, it also saved your resolution. Now undo the merge and do it again, as you would when a rebase or a test merge brings back the same conflict:

Git still stops (you should check its work), but the file already contains your earlier resolution. rerere.autoUpdate true also stages it. git rerere forget <path> deletes a recorded resolution that was wrong, and git rerere status shows what rerere is tracking.

Submodules and Subtrees

Sometimes a project needs code from another repository: a shared theme, a library you cannot install as a package, configuration shared by several services. Git has two ways to include it.

Submodules: a pointer to a commit in another repository

A submodule is a repository inside your repository. Yours records only its URL (in .gitmodules) and the exact commit to use. The team has a separate brand repository with shared colors:

The first attempt failed on purpose: since Git 2.38.1, submodules may not be cloned from local paths unless you allow it, because a malicious repository could otherwise read files from your disk. With HTTPS or SSH URLs, which is what you use in real projects, the -c protocol.file.allow=always is not needed.

A plain git clone gives you an empty vendor/brand directory. Clone with the submodules, or initialize them afterwards:

For a clone that already exists: git submodule update --init --recursive. When the brand repository gets a new commit, nothing changes in your project until you move the pointer:

Submodule rules that save time:

• The submodule is checked out at a commit, in detached HEAD. To change its code, cd into it, switch to a branch, commit and push there, then commit the new pointer in the parent.
• git config set --global submodule.recurse true makes pull, switch and checkout update submodules automatically.
• Everyone who clones needs access to every submodule URL.
• Removing a submodule takes several steps (remove a Git submodule).

Subtrees: a copy of the other repository's files

git subtree copies another repository's files into a directory of yours, as normal files in normal commits. Nobody who clones needs to know about it. It ships in Git's contrib directory and is included in most packages:

git subtree pull --prefix=vendor/brand-copy <url> main --squash brings in later changes, and git subtree push sends changes made in your copy back.

	Submodule	Subtree
What your repository stores	A URL and a commit hash	The files themselves
Cloning	Needs --recurse-submodules or an extra command	Nothing special
Pinning a version	Exact, by commit	By what you last pulled
Changing the shared code	In the other repository, then update the pointer	Directly, then subtree push if you want to share it
Good for	Large or separately released dependencies, strict pinning	Small shared code that you rarely update

If the other project publishes packages (npm, PyPI, Go modules, Helm charts, Terraform modules), use the package manager instead of either.

Useful Attributes: Diff Drivers and Export Rules

Besides line endings (Part 9), .gitattributes controls how Git treats paths:

Attribute	Effect
export-ignore	Leave the path out of git archive (release tarballs)
diff=<driver>	Diff the file through a converter that prints text, for example *.docx diff=docx with git config diff.docx.textconv "pandoc --to=plain"
-diff	Show "Binary files differ" instead of a diff (for minified or generated files)
merge=ours	Keep your version when both sides changed the file and Git has to merge its content (needs git config merge.ours.driver true); changes made only on the other side still come in
linguist-generated	GitHub hides the file in pull request diffs and language statistics
filter=lfs	Set by git lfs track (Part 9)

git check-attr -a <path> shows the attributes that apply to a file.

Archives and Bundles

git archive writes the files of a commit or tag as a tar or zip file, without .git: the usual way to make a source release. Leave the docs out of release archives first:

The submodule directory is in the archive but empty: git archive does not include submodule contents.

A bundle is a whole repository (or part of one) in a single file, which you can clone, fetch from or verify like a remote. It moves history where there is no network between the two machines, or makes an offline backup:

Script Git Safely

Commands such as git status and git branch are meant for people, and their output can change between versions. In scripts and CI, use the plumbing commands and machine-readable formats, which Git keeps stable:

• --porcelain output (git status, git worktree list) is a stable format. Add -z to it, and to git ls-files and git diff --name-only, to separate entries with NUL bytes, so file names with spaces or newlines cannot break your parsing. git blame --porcelain has its own stable, line-based format.
• git rev-parse, git for-each-ref, git ls-files, git cat-file --batch and git merge-base --is-ancestor A B (exit code 0 if A is an ancestor of B) answer most questions without parsing human output.
• In CI, a shallow checkout (fetch-depth: 1 in GitHub Actions) has no history: git describe, git diff origin/main...HEAD and changelog tools fail or give wrong results. Fetch the history you need, or use a partial clone (--filter=blob:none), which is fast and complete. The CI/CD guide's section on faster and cheaper runs covers checkout depth in practice.

Refspecs: how fetch and push map branches

When you clone, Git writes a refspec into the remote's configuration. It maps branches on the remote to your remote-tracking branches:

+refs/heads/*:refs/remotes/origin/* means "every branch on the remote, stored as origin/<name> here; + allows non-fast-forward updates". You can add more. For example, GitHub publishes every pull request as refs/pull/<number>/head, and this extra line fetches them all as origin/pr/<number>:

git config set --append remote.origin.fetch "+refs/pull/*/head:refs/remotes/origin/pr/*"
git fetch origin
git switch --detach origin/pr/123

Push takes refspecs too: git push origin HEAD:refs/heads/test pushes your current branch as test, and git push origin :old-branch (an empty source) deletes old-branch.

Common Mistakes

• A submodule pointer committed without pushing the submodule commit. Others then cannot check out the commit your project points to. Push the submodule first (git push --recurse-submodules=check refuses otherwise).
• Deleting a worktree folder by hand. Use git worktree remove, or run git worktree prune afterwards.
• Shallow clones where history matters. Use a partial clone instead.
• Parsing git status or git branch output in scripts. Use --porcelain formats and plumbing commands.
• SHA-256 for a project that will move to GitHub. It cannot be pushed there yet.

Summary

• Git stores blobs, trees, commits and tags, named by the hash of their content; refs are names for hashes.
• SHA-256 and reftable work today and become defaults in Git 3.0, but SHA-256 repositories cannot exchange commits with SHA-1 ones.
• Worktrees give you several branches checked out from one repository.
• Partial clones, sparse checkout and scalar make large repositories manageable.
• rerere remembers conflict resolutions; submodules pin another repository's commit, subtrees copy its files.
• git archive, bundles, porcelain formats and refspecs cover releases, offline transfer and scripting.

Where to Go Next

You have used every part of the Git workflow on one project: from git init to a team with pull requests, signed releases, hooks and a history you can search. Next steps:

• Put the checks into a pipeline with the Introduction to CI/CD guide.
• Test yourself with the Git Command Quiz, the Git Fundamentals flashcards and the Git Concepts simulator.
• Practice for interviews: Git staging and committing, rebase vs merge.
• The free Introduction to Git and GitHub eBook covers the basics again from another angle.

Keep Learning

• A repo per agent: what Cloudflare Artifacts changes about Git
• Clone a specific folder of a repository
• A poisoned .git/config runs code on git status