Context compression beyond summaries

A long-running agent accumulates history fast: tool outputs, file contents, intermediate reasoning, prior turns. Eventually it exceeds the window and something has to give. The default move, send it all to the model and ask for a summary, has three problems. It costs a full model call every time you compress. It adds latency in the middle of a task. And LLM summaries are lossy in exactly the wrong way: they preserve the narrative and drop the specifics, the exact error string, the precise line number, the flag value, which is the information the agent actually needed. Summarization has its place, but it should be the last tool you reach for, not the first.

1. Extractive compression: keep the lines that matter, verbatim

Most of what fills a context window is low-value text surrounding a few high-value lines. A 500-line test log is mostly dots and passes; the signal is the three lines of the failing assertion. Extractive compression keeps the important lines exactly as they are and drops the rest, with no model call, no paraphrasing, and no lost specifics. Rules that pay for themselves immediately:

• In tool output, keep error and warning lines, stack frames pointing at project code, and the final status; drop progress and success noise.
• In file contents already shown once, replace repeat views with the changed hunks only.
• In command output, keep the head and tail and elide the middle with a marker rather than a paraphrase.

2. Structural compression: replace payloads with handles

The second technique treats large objects as references rather than inline text. When a tool returns a 4,000-token JSON blob, the agent rarely needs all of it in context. It needs to know the blob exists, its shape, and how to get a specific field later. Replace the payload with a schema plus a handle: search returned 30 results with id, title, and score; ask for any by index. The agent can request the exact rows it wants when it wants them. The same idea covers deduplicating repeated content, collapsing a file you have shown to a RepoMap-style outline, and dropping tool arguments once the result is in hand.

The question is never how do I summarize this? It is does the agent need these tokens in context right now, or does it need to know they exist and how to fetch them?

3. Learned token pruning: when it is worth it

The third family drops individual low-information tokens using a small model to score them, the approach behind methods like LLMLingua. It can compress prose two to three times with little quality loss, and unlike summarization it does not hallucinate. The tradeoff is that it needs a scoring pass and tuning, so it earns its place on stable, high-volume prompt sections, long system instructions and reference documents, rather than on the volatile tail of a conversation.

Eviction: decide what never gets dropped

Whatever technique you use, you need an eviction policy, an explicit order of what to compress first. A policy that holds up in practice: never touch the system prompt or the current task; always keep the most recent turn or two intact; compress or evict the oldest, largest, lowest-relevance items first. Keep a durable scratchpad of decisions and facts the agent has committed to, so compressing the raw history that produced them is safe.

Measure it like retrieval

Compression is a retrieval problem in disguise: did the compressed context still contain what the next step needed? Track task completion before and after compression is enabled, and where you can, context recall, of the facts the agent later needed, how many survived. If completion holds and cost drops, the compression is working; if completion dips, you compressed away signal, and the fix is a better eviction policy, not a bigger summary.