← all posts

Context Is a Budget, Not a Bucket

4 min read #llm-tooling

The cheapest token is the one never emitted. Every other optimization in LLM agent tooling — caching, batching, model routing — is fighting over cents, while tool outputs quietly dump thousands of tokens of noise into a context window that is the model's entire working memory. Context is not a bucket you fill; it's a budget you spend, and every tool you give an agent is a program that spends it on the model's behalf, usually badly.

I learned this building agent tooling for my own projects — a local agent with file search, shell, and a knowledge base behind MCP servers. The first version had beautifully "complete" tools: search returned every match with full lines, the shell returned entire outputs, list operations returned everything. The agent got stupider as sessions went on, in a very characteristic way: it re-read things it had already seen, lost track of its own plan, and started pattern-matching on irrelevant text that happened to be sitting in context. Nothing was wrong with the model. I was strip-mining its working memory.

Two costs, one of them invisible

A wasted token costs you twice. The visible cost is dollars and latency — linear, boring, survivable. The invisible cost is attention: every irrelevant line in context is a distractor the model must implicitly decide to ignore on every subsequent step, and models are measurably worse at recall and reasoning as relevant signal gets diluted. A 5,000-token log dump doesn't just cost 5,000 tokens once; it taxes the quality of every decision made after it. That's why the budget framing matters: the question is never "can this fit," it's "what am I displacing."

Tools should answer questions, not dump state

The core design mistake is building agent tools like APIs for programs, which want completeness, instead of like reports for a colleague, who wants the answer. Compare two returns for the same search over a repo:

# version 1: "complete" — 2,000 matches, ~40k tokens, useless
src/a.py:12: from config import load
src/a.py:97: cfg = load(path)
… 1,998 more lines …

# version 2: shaped — ~200 tokens, actionable
2,014 matches in 143 files. By directory:
  src/core/     812   src/api/  411   tests/  605   …
Top files: src/core/config.py (94), src/core/init.py (61)
Showing first 10 matches; refine query or pass dir= to drill down.

Version 2 costs 200× less and is more useful, because it answers the question the agent actually had ("where does this concentrate?") and hands it a lever for the follow-up. The design rules that consistently pay off:

  • Truncate with structure, not with scissors. Never chop at N bytes and append "…truncated". Return head + tail (errors live at the ends of logs), plus a count of what was omitted: [1,847 lines omitted — 212 similar warnings, 3 distinct errors]. The summary of the cut is worth more than another screen of the middle.
  • Deduplicate aggressively. A build log with the same warning 200 times should reach the model as one line with a count. Compilers repeat themselves; the context window shouldn't.
  • Return handles, not payloads. When output is genuinely large, write it somewhere and return an ID plus a summary: saved to /tmp/log-3f2a (18k lines); errors at lines 1204, 9917. The agent that needs the details can page in exactly the slice it wants. This is pointers versus values, rediscovered.
  • Make drill-down cheap and say it exists. Truncation without an escape hatch teaches the model that information is gone. The line "pass offset= to see more" is what converts a lossy tool into a lazy-loading one.

Descriptions are the tax you pay every single call

Tool outputs at least only spend budget when invoked. Tool descriptions are in context on every request, forever. Twenty tools with lovingly detailed 400-token descriptions is an 8,000-token standing charge before the conversation begins — and most of it is prose the model doesn't need, restating parameter names or explaining what "recursive" means. Write descriptions like man-page NAME lines, not man pages: one sentence of purpose, parameter constraints the schema can't express, one example only if the call shape is genuinely surprising. If a tool needs three paragraphs to describe, that's usually two tools wearing a trenchcoat — or a tool whose interface leaks its implementation.

The same standing-charge logic applies to the roster itself. Every tool the agent will not use in this session is pure distractor. Registries that expose 50 tools "to be safe" are spending budget to lower the probability that the right tool gets picked — tool-selection accuracy degrades as the menu grows. Scope the toolset per task, or expose a search-then-load pattern where the model pulls in schemas on demand.

The test

My working heuristic now: read every tool result out loud to an imaginary colleague. If you'd be embarrassed to recite it — if you'd instinctively skip, summarize, or say "long story short" — the tool is making the model do that compression instead, at your expense, on every turn. The model can do that compression; it's remarkably good at it. But you're paying it to reason about your problem, not to be a garbage compactor. Spend the engineering hour once, in the tool, and the model spends its attention where you wanted it all along.