If you drive an AI coding agent all day — Claude Code, Antigravity, Codex CLI, Pi / OMP, Aider — you know the shape of the stall. Turn 1 feels like magic. Somewhere around turn 8, you type a one-line follow-up and the terminal does nothing for a long time: fans spin up, a local GPU allocates tens of gigabytes of VRAM, a cloud bill ticks upward, and then a single character appears.
The usual advice is to run /compact, or to let the harness auto-summarise older turns. That advice is how you get the stall back.
This article is the systems view of the problem: why agent context is an append-only memory leak, why LLM summarisation is the wrong garbage collector, and what the numbers look like when you fix it deterministically instead. The fix is contextWatcher (cw), an MIT/Apache-2.0 context compiler and intercepting proxy written in Rust 2024.
The append-only memory leak
LLM APIs are stateless HTTP endpoints. An agent “remembers” by re-uploading the whole conversation as one JSON array, every single turn:
Turn 1: system + tools + prompt -> 5,000 tokens
Turn 2: turn 1 + output + tool result -> 12,000 tokens (turn 1 re-sent)
Turn 3: + file reads + build logs -> 28,000 tokens (everything re-sent)
Turn 10: 100,000+ tokens re-sent on every keystrokeTwo things break as that array grows.
Attention stops being uniform. Long-context models attend most strongly to the beginning (primacy) and the end (recency) of a prompt, and least to the middle — the U-shaped curve documented by Liu et al. in Lost in the Middle. Your AGENTS.md rules (“preserve all existing docstrings”, “run the linter before committing”) sit at the front and survive; the mid-session constraint you gave at turn 4 does not. The agent then re-reads files it already read and re-introduces bugs it already fixed.
Prefill becomes the bottleneck. Prompt processing is compute-bound and grows with sequence length, and every token in the prompt needs a KV-cache slot in VRAM. If the cached prefix is intact, the engine only evaluates the new tail and the first token arrives in milliseconds. If the prefix changed, the engine re-evaluates every token from index zero before it can emit anything. That is Time To First Token (TTFT), and it is where the multi-minute stalls come from.
Why /compact makes it worse
Compaction sends your whole conversation to a model with a prompt like “summarise the above in 200 words” and replaces the history with the result. It shrinks the token count and costs you four things:
- It destroys the KV-cache prefix. The summary replaces turns 1–20, so token zero itself changes. Every cached block is invalid. You keep the shorter prompt and pay a full cold prefill for it, plus you lose the prompt-caching discount that cloud providers price in.
- It is lossy in exactly the way code can’t afford. “Refactored the parser regex” does not contain the error trace, the line numbers, or the function signature the agent needs two turns later. The model fills the gap by guessing.
- You pay completion tokens to save input tokens. Compaction is itself an inference call that reads 30k tokens and waits 15–30 seconds before your real turn starts.
- It cannot see structure. It does not know that
src/auth.rsappears four times in the window, that three of those copies are superseded, or that 3,890 lines of compiler output came from a test run that has since passed. It summarises prose blindly, so it keeps noise and discards ground truth.
Treating context like virtual memory
The interesting question is not “how do we summarise a transcript”. It is: what if context were an address space with deterministic garbage collection?
cw sits between the harness and the model as an intercepting HTTP/SSE proxy. It parses each request into a canonical context graph — turns, nodes, tool calls, file reads — while keeping a lossless JSON-pointer map back to the raw payload, so a rewrite touches only the JSON paths it targets and provider-specific fields (Anthropic thinking signatures, Gemini thoughtSignature, cache-control headers) pass through untouched.
Rewrites are stored as content-hash-keyed overlays, not as edits to a mutable transcript. Because the harness re-sends its own unmutated history on every turn, the overlay is re-applied automatically on each subsequent request; nothing about it depends on the previous request having been mutated. Four operators do the work, all deterministic, no model calls, no tokens spent:
| Operator | What it does | Effect |
|---|---|---|
| Shadow dedupe | When a file is read again after being modified, the superseded copy is replaced with a zero-token tombstone that keeps the path and a hash; the newest copy stays byte-for-byte intact | Measured in the run below: 6 copies in the window → 1 live copy + 5 stubs (−97% of those tokens) |
| Log pruning | Keeps the tool call id, the command, the first lines, and the exit code; replaces the repetitive middle of a failing build log | Thousands of repeated lines become one stub line, with the tool schema intact |
| Tail rule pinning | Mirrors critical constraints from the attention trough directly above the latest prompt, without touching the cached prefix | Architectural rules stay in the high-salience zone |
| MCP schema gating | Tracks which MCP servers are actually used and gates the unused ones out of the tool block | Reclaimed tokens scale with how many servers you attach; nothing is re-sent for a server you never called |
The design contract from the repository’s own AGENTS.md is that proxy overhead stays under 2 ms; that is a budget the implementation is held to, not a number I am claiming as a benchmark here.
Measured on real silicon
Benchmarks that need a cloud account to reproduce are hard to trust, so this ran on hardware we own: a dual AMD Radeon Instinct gfx906 workstation running Qwen3.8-Flash-Next (IQ2_XS, 131k context) on a local llama-server, driven by a real agent harness building a crash-safe key-value store with a WAL, MemTable, snapshots, and a client pool. One workflow, 3 turns, 33 context nodes, and a real 8-test unit suite as the correctness oracle. Every number below is from the committed artifact benchmark/data_longrun/benchmark_longrun_matrix.json.
| Configuration | Context tokens | Saved | KV prefix intact | TTFT | Unit tests |
|---|---|---|---|---|---|
| Unmanaged harness | 27,454 | — | 0.0% (cold) | 171,587 ms | 8/8 |
cw optimize min (cache-safe) |
27,077 | 377 (1.4%) | 100.0% | 0 ms (cache hit) | 8/8 |
cw optimize mid (surgical) |
14,488 | 12,966 (47.2%) | 98.2% | 1,630 ms | 8/8 |
cw optimize max (deep fold) |
14,206 | 13,248 (48.3%) | 96.5% | 3,108 ms | 8/8 |
Harness /compact |
950 | 26,504 (96.5%) | 0.0% (wiped) | 5,938 ms | 0/8 |
Three things in that table are worth more than the headline number.
The speedup is 105×, and it is a ratio, not a constant. 171,587 ms down to 1,630 ms at the mid tier is 105.3× on this host with this model. The absolute 171 seconds is a property of the gfx906 box running a 27k-token prefill uncached; on a cloud provider the same mechanism shows up as a bill and a 3–5 second wait instead of three minutes. What transfers is the mechanism: keep the prefix, skip the recompute.
min reports 0 ms because the prefix was still in cache. That is not a free lunch — it is the same measurement taken when the cached blocks were still resident, which is precisely the state a prefix-preserving rewrite leaves you in. If the cache has been evicted, min pays a normal prefill.
More compression is slower, not faster. max keeps 48.3% of the tokens but prefill is two times the mid tier’s, because folding turns 1–2 into one node changes 3.5% more of the prefix (96.5% intact versus 98.2%) and the engine has to recompute from the first changed token. Aggressive rewriting that ignores the cache boundary is how well-intentioned context tooling ends up slower than doing nothing. mid is the tier we default to.
And the /compact row is the argument of this article in one line: it removed 96.5% of the tokens, still paid a 5.9 s cold prefill on a 950-token prompt, and lost the ground truth the model needed to pass any test.
Multi-agent swarms
The same prefix logic repeats across agents. Four subagents working the same repository each re-send the same ~3,200-token system prompt, tool block, and repo rules. Routing them to a shared trunk in GPU memory (benchmark/artifacts_swarm/SWARM_BENCHMARK_REPORT.md):
| Metric | Unmanaged | cw swarm |
|---|---|---|
| Total team prompt tokens | 12,886 | 3,286 (−74.5%) |
| KV cache in VRAM | 3,221 MB | ~2.4 GB saved |
| Team prefill, trunk seed included | 85.6 s | 21.2 s (4.0×) |
| Follow-on subagent TTFT | 20.7–21.7 s | 127–173 ms (123–172×) |
Note the two different ratios: the follow-on agents get a 100×-class speedup because they hit a warm shared trunk, while the team total is 4.0× because the first agent still pays to seed it. Quoting the smaller number as if it were the large one is the kind of thing benchmark tables are usually built to hide.
Beyond that, the prototype also splices KV pages at engine level — dropping 300 aligned 16-token pages (4,800 tokens, ~1.2 GB) from the middle of a live session with zero prefix invalidation — and renders a positional attention “ECG” that shows when critical instructions are sinking into the low-salience middle.
What these numbers do not prove
Being precise about the denominator matters more than the multiplier:
- One workflow, one model, one host. 3 turns, 33 nodes, Qwen3.8-Flash-Next IQ2_XS on dual gfx906. This is a measured case, not a fleet average. Long sessions with heavy tool output should compress better; a two-turn chat barely at all.
- The unit-test oracle is the strongest claim here. 8/8 passing after
midmeans the rewrite did not remove information the task needed. It does not mean no rewrite can ever hurt a task. - No prebuilt binaries are published yet. The installer detects your platform (Linux and macOS, x86_64 and aarch64), tries the release tarball, and falls back to
cargo install --git— which needs a Rust toolchain and a few minutes of compiling. The fallback path works today; the tarball path will be the fast one once releases are cut. cwnever calls a model to compress. All four operators are deterministic rewrites — which is why they cost $0.00 and cannot hallucinate — but “deterministic” also means they follow rules, not judgement.maxfolding decisions are structural, not semantic.
Try it
Linux or macOS, no root, installs to ~/.local/bin/cw, adds it to your $PATH if needed, and registers plugins, skills, and slash commands for the harnesses it finds (Antigravity, Claude Code, Codex CLI, Pi / OMP, Hermes):
curl -fsSL https://constant-labs.com/cw-install.sh | shRead the script before you pipe it into a shell — the whole installer is under 250 lines of POSIX sh — or build from source:
git clone https://github.com/GionaGranchelli/context-watcher.git
cd context-watcher
cargo install --path crates/cw-cli
cw initThen watch what it saves:
cw gain # token, KV-cache and dollar savings scoreboard
cw hud # sub-millisecond shell statusline
cw pulse # ambient background savings notifications
cw status # current session context and VRAM footprintSource, specs, and every benchmark artifact are public at github.com/GionaGranchelli/context-watcher under MIT OR Apache-2.0.
FAQ
Does /compact make an AI coding agent faster?
It makes the prompt shorter but usually the first token slower. Harness compaction replaces the start of the conversation with a new summary, which invalidates the KV-cache prefix from token zero. In our measured run the model went from an uncached 171.6 s prefill to a still-uncached 5.9 s prefill on a near-empty context, and failed all eight unit tests because the exact function signatures were gone.
What is a KV-cache prefix, and why does changing old turns break it?
A transformer caches attention state for prompt tokens sequentially from index zero. If any early token changes, every cached block after it must be recomputed. Rewriting turns 1–20 into a summary paragraph therefore throws away the whole cache, while a change confined to the tail keeps everything before it valid.
Does contextWatcher send my context to a cloud LLM to compress it?
No. cw compiles the prompt locally with deterministic, content-hash-keyed rewrite rules and makes no model calls, so compaction costs zero tokens and zero dollars in inference. Your transcript never leaves the machine.
