Why Build Caches Miss

A build cache almost never misses because it's too small. It misses because you lied to it. Content-addressed caching is a very simple contract: the key is a hash of every input to an action, and the value is the output. If two machines compute different keys for what a human would call "the same build," the cache didn't fail — one of the inputs was something you didn't think of as an input. Every mysterious miss I have ever debugged, across CMake, Make, and Bazel projects, came down to that sentence.

I learned this properly on a personal project: a C++ toolchain I built for a hobby renderer, with a shared remote cache between my desktop and laptop. Same repo, same commit, same compiler version — and a hit rate of roughly 40%. Here is the taxonomy of everything that turned out to be lying, because it's the same list you will find in any codebase.

The key is the whole environment, whether you like it or not

A cache key for a compile action is conceptually:

key = hash(
  command line,
  contents of every input file,
  compiler binary,
  environment the action can observe
)

That last line is where builds die. If your build system passes the full environment through to actions — and plain Make and most hand-rolled CI scripts do — then PATH, LANG, TZ, HOME, and whatever your shell profile exports are all silently part of the key. Two agents with different PATH orderings produce different keys for identical work. Bazel's answer is --incompatible_strict_action_env and explicit --action_env allowlisting: actions see a fixed, minimal environment, and anything else must be declared. If your build tool can't do that, wrap actions in env -i with an explicit allowlist. It feels paranoid until you diff two "identical" CI runs and find the key differed because one runner had LANG=en_US.UTF-8 and the other LANG=C — which changes sort order, which changes a generated header, which changes everything downstream.

The classic sins, in the order you'll find them

Timestamps in outputs. __DATE__ and __TIME__ in C/C++ are the famous ones — every rebuild produces a byte-different object, so nothing downstream ever hits. GCC and Clang both warn with -Wdate-time, and both honor SOURCE_DATE_EPOCH, which pins them to a fixed instant:

# make "now" a function of the commit, not the wall clock
export SOURCE_DATE_EPOCH=$(git log -1 --format=%ct)

Absolute paths in debug info. Build the same file from /home/you/src and /builds/agent-7/src and the DWARF sections differ, so the objects differ, so the cache key of every consumer differs. The fix has been in compilers for years and almost nobody passes it:

-ffile-prefix-map=$PWD=.   # covers __FILE__ and debug paths
-fdebug-prefix-map=$PWD=.  # older compilers, debug info only

Bazel sidesteps the whole class by executing every action inside a sandbox at a stable, fake working directory. That's not sandboxing for security — it's sandboxing so the path can't leak into the artifact.

Archive nondeterminism. tar and zip happily embed mtimes, uids, and whatever order readdir() felt like returning today — which differs between ext4 and tmpfs, so your laptop and CI disagree forever. Deterministic archiving is one line once you know to write it:

tar --sort=name --mtime=@0 --owner=0 --group=0 --numeric-owner \
    -cf out.tar dir/

ar and linkers. Static archives embed timestamps unless you use deterministic mode (ar rcD, or a binutils built with --enable-deterministic-archives). Some linkers embed a build ID derived from the link time rather than the content. ld --build-id=sha1 makes it a function of the bits instead.

Hash-map iteration order. Any code generator that walks a hash map and writes results to a file is a nondeterminism machine. Go randomizes map order on purpose; Python dicts are insertion-ordered but sets are not. The rule for anything that generates code, serializes config, or writes a manifest: sort before you emit. Always. It costs an O(n log n) you will never measure and buys you a cache that works.

Prove it, don't vibe it

The test for all of this is brutally simple: build twice, in two different directories, and compare the bits.

# the two-build test
git clone . /tmp/a && git clone . /tmp/b
(cd /tmp/a && ./build.sh) && (cd /tmp/b && ./build.sh)
diff <(cd /tmp/a/out && sha256sum *) \
     <(cd /tmp/b/out && sha256sum *)

When hashes differ, diffoscope will tell you exactly which bytes and usually why — it understands ELF sections, archive members, and zip metadata, so "the .debug_str section contains /tmp/a" pops right out. On my renderer, four fixes — strict action env, prefix maps, SOURCE_DATE_EPOCH, and one sorted dict in a code generator — took the cross-machine hit rate from ~40% to 97%. The remaining 3% were genuine source changes.

Why this compounds

The vicious part of cache misses is that they propagate forward. One nondeterministic object file at the bottom of the graph re-keys every action above it: the archive that contains it, the binary that links it, the test that runs the binary, the image that packages it. A single __TIME__ in a low-level utility library can, by itself, force a near-full rebuild of everything, every time — and the profile will just say "cache miss," not "cache miss because of one macro three layers down." That's why hit rate is a graph property, not a file property, and why fixing the lowest-level offenders first pays off disproportionately.

Hermeticity has a reputation as build-system zealotry. It isn't. It's just the discipline of making the cache key honest — making the set of things that can affect the output equal to the set of things the build system knows affect the output. Once those two sets match, caching isn't a feature you tune. It's just what hashing does.