Context Compression Lab
The support agent is cheap and fast because the context was compressed aggressively. The hidden failure: it no longer remembers enough policy, account state, and order evidence to make a reliable refund decision.
Context compression is not a contest to minimize tokens; a defensible policy removes low-value context while preserving the instructions and evidence that determine the task outcome.
The original working set contains 25,500 synthetic tokens. Production allows a 16,000-token working context. Start from a policy that saves more than two thirds of the tokens, then find out what those savings removed.
—
—
—
—
—
—
—
—
Token retention and semantic retention are not identical. A deeper summary can preserve more useful meaning per retained token, while a retrieval or memory budget can still exclude information before summarization gets a chance to protect it.
| Context segment | Original tokens | Active tokens | Semantic retention |
|---|
The initial policy aggressively minimizes the working context and silently discards task-critical information.
Why can a shorter context make an AI system worse even when the summary sounds correct?
A language model does not receive the original application state directly. It receives a constructed working context: instructions, the user request, retrieved evidence, memory, tool observations, and background material. Compression changes that constructed environment. A fluent summary can therefore sound reasonable while omitting the one account flag, policy exception, or prior promise that changes the correct action.
Compression ratio is only the visible knob
Teams often discuss compression as a percentage: reduce the context by 50%, 70%, or 80%. That number is not enough. Two policies with the same token count can preserve very different information depending on which segments were summarized, what the retrieval layer admitted, and which facts were protected from lossy transformation.
Critical facts deserve a different policy from background context
In this scenario, the system and safety policy, current task, account eligibility, refund policy, and order evidence are marked critical. Turning on critical-fact protection reserves more of their representation before less important conversation history and product background. This is a simplified model of a common production pattern: classify information by consequence before applying a generic compressor.
Retrieval and memory budgets happen before generation
A low retrieval budget can starve the model of evidence even when the compression policy protects critical facts. The missing evidence cannot be recovered by a better prompt or a deeper summary because it never entered the context. Memory has a similar trade-off: too little can remove useful continuity, while too much can crowd the working set with stale or weakly relevant history.
Why does the Lab include a 16k budget?
Without a hard capacity constraint, the easiest way to maximize simulated quality would be to retain nearly everything. Real systems have context limits, latency targets, and unit economics. The 16,000-token budget makes the opposite failure visible: an under-compressed policy may retain perfect information but still be operationally invalid.
What is simulated here?
All token counts, semantic-retention values, quality scores, hallucination-risk values, latency indexes, and cost indexes are deterministic educational quantities. They are not measurements from a live model, tokenizer, summarizer, retrieval system, or customer-support workload. Production thresholds must be established with representative tasks, real model behavior, and evaluation evidence.
Common questions
Protecting consequential instructions and evidence is a strong default, but real systems still need task-specific classification, freshness rules, and conflict resolution.
No. It can preserve more meaning but also consumes more tokens and processing. The right depth depends on the task and budget.
A larger model context can reduce immediate truncation pressure, but it does not remove retrieval quality, stale memory, latency, attention allocation, or cost trade-offs.
Prompt compression is one technique inside a broader context-engineering problem. The working context also includes retrieval, memory, tool state, conversation history, and application-owned instructions.
Define a context policy for a refund-capable support agent.
Choose what must survive compression, how much evidence retrieval can admit, how much memory is worth carrying, and where you would set the context budget. Explain which information you are willing to lose and which information must be release-blocking if absent.