Long Context Has One Villain, and It Is the Cache
It grows with every token, it never shrinks on its own, and it eats your GPU memory until you either buy a bigger card or start throwing tokens overboard. Throwing tokens overboard is the interesting option. The question that actually matters, the one everyone skips, is which ones.
The dumb answer, keep the newest and forget the rest, works great right up until the question depends on the one thing you already forgot, at which point the model just makes something up and says it with total confidence. The research answer is to keep the tokens that actually mattered, using signals like accumulated attention (H2O, Zhang et al. 2023) or a small set of protected attention-sink tokens (StreamingLLM, Xiao et al. 2023).
kvsqueeze puts those policies side by side on a task where forgetting is measurable, and shows exactly how far each one compresses before recall falls off a cliff.
A Needle You Can Actually Watch Get Evicted
The benchmark streams a long sequence through a fixed-budget cache. One early token is the needle a later query needs. A policy only passes if that token is still alive when the query arrives.
• Recency window: keeps the newest tokens, drops the oldest. Strong until the answer ages past the window.
• Heavy hitter: keeps whatever has actually been attended to the most, drops the rest. This is the H2O idea.
• Attention sink: always protects the first few tokens plus a recent window. This is the StreamingLLM fix for the collapse a pure recency window hits.
The Curve, Not Just One Number
A single budget point is a footnote. The real deliverable is the survival curve as you sweep the budget down from generous to brutal. At 80% of full cache every policy looks fine. By 24% the simple policies have flatlined at zero recall while heavy hitter is still holding on.
The Number
At 40% of full cache, heavy hitter holds 57% needle recall. Plain recency manages 15% and is basically checked out mentally by that point. Same memory budget, nearly a 4x gap in what actually survives. Push down to 10% of full cache and heavy hitter is the only policy still recalling anything at all, the last one still awake at the meeting.
The Last One Awake Was Actually Asleep the Whole Time
Every needle in that curve lives somewhere in the first 70% of the sequence, and gets asked about only after the whole stream finishes. Nothing in the benchmark ever asks "do you remember what I just said." So I asked it myself: I queried for something from the last 15 tokens of a 400-token stream instead. Heavy hitter's recall of anything recent was 0%. At every single budget from 10% to 80%. Not degraded, not brittle, structurally incapable.
The mechanism is almost embarrassingly simple once you see it. Heavy hitter evicts whichever live token has the least accumulated attention, ties broken toward the older token. Every token in this simulation only ever earns attention from a shared local-recency signal the moment it's near the write head; once that passes, the credit is frozen for good. A token that was just appended hasn't gotten any of that credit yet, so it starts at zero, strictly below every token that already made it through the window. Eviction runs the instant a token is appended, so the newest arrival is always the weakest candidate the moment the cache is full. It gets evicted on the spot. Every time. The cache doesn't compress under pressure, it just stops working the moment it fills up.
I went looking for the mechanism the README credited for the win, "later queries re-attend to the needle, giving heavy hitter the signal it's built to exploit," and deleted it entirely from the benchmark. Zero of 40 trial outcomes changed, at every budget tested. The signal that was supposed to be doing the work was inert. The real win came from the freeze, which looks exactly like perfect recall on a benchmark that never once asks about anything after the initial fill.
The actual H2O paper doesn't have this problem, because it never relies on pure attention ranking alone: it splits the budget between a heavy-hitter pool and an unconditionally protected recent window, so a new token gets a grace period to prove it matters before eviction pressure hits it. That's the piece this implementation was missing. Adding it back fixes recent recall completely, 0% to 100% at every budget, and as a bonus strictly beats the original on the exact benchmark above too: 100% recall at every budget from 10% to 80%, instead of the original's 12% to 100%. Confirmed on a held-out set of positions and seeds I evaluated exactly once. The original benchmark file is untouched, so the published curve above still reproduces exactly.
19 tests pass in CI across Python 3.9, 3.11, and 3.13, all deterministic.