Echo:无KV缓存的关联回忆与谱Koopman算子
Echo: KV-Cache-Free Associative Recall with Spectral Koopman Operators
AI总结:
Echo通过谱Koopman算子实现无KV缓存的关联回忆,解决了传统Transformer的内存瓶颈问题,在多个基准测试中展现出卓越的检索性能。
摘要:
长链式推理和代理工具调用会产生数万token的轨迹,但Transformer的KV缓存随序列长度线性增长,导致在商用硬件上出现内存瓶颈。状态空间模型提供常数内存递归但存在内存悬崖:一旦存储事实与查询之间的差距超过递归状态的有效范围,检索准确性就会崩溃。我们引入Echo,一种无KV缓存的关联回忆架构,围绕Spectral Koopman Attention(SKA)构建;SKA是一种可直接替换注意力层的算子,它通过闭式动力学算子增强SSM块,其充分统计量在常数内存中累积,无需KV缓存。Echo通过核岭回归拟合键值历史的谱线性系统,并通过学习的幂迭代滤波器进行检索,所有操作都在O(r²)流式状态中完成,其中r是小投影秩。在多查询关联回忆基准测试中,纯Mamba-2 SSM在所有间隙长度和KV对数量上都无法超过偶然准确性(约3%),而SKA增强模型在5000万参数规模下,每种配置测试均达到100%检索准确率,包括有4096个token的干扰间隙和32个KV对。在五个额外的迁移基准测试中,包括针尖在 haystack、工具轨迹和多跳检索,SKA始终优于纯SSM和SSM+注意力混合模型,同时保持常数推理内存。消融实验确认,谱算子而非前缀掩码策略驱动了检索增益。
英文摘要:
Long chain-of-thought reasoning and agentic tool-calling produce traces spanning tens of thousands of tokens, yet Transformer KV caches grow linearly with sequence length, creating a memory bottleneck on commodity hardware. State-space models offer constant-memory recurrence but suffer a memory cliff: retrieval accuracy collapses once the gap between a stored fact and its query exceeds the effective horizon of the recurrent state. We introduce Echo, a KV-cache-free associative recall architecture built around Spectral Koopman Attention (SKA); a drop-in replacement for attention layers that augments SSM blocks with a closed-form dynamical operator whose sufficient statistics are accumulated in constant memory with no KV cache. Echo fits a spectral linear system to the key and value history via kernel ridge regression and retrieves through a learned power-iterated filter, all from $O(r^{2})$ streaming state where $r$ is a small projection rank. On the Multi-Query Associative Recall benchmark, a pure Mamba-2 SSM fails to exceed chance accuracy (${\sim}3\%$) across all gap lengths and KV-pair counts, while at the 50M parameter scale SKA-augmented models achieve $100\%$ retrieval accuracy on every configuration tested, including distractor gaps of $4{,}096$ tokens with $32$ KV pairs. Across five additional transfer benchmarks including needle-in-a-haystack, tool-trace, and multi-hop retrieval, SKA consistently outperforms both pure SSM and SSM+Attention hybrids while maintaining constant inference memory. Ablations confirm that the spectral operator, not the prefix masking strategy, drives the retrieval gain.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。