查询知道该遗忘什么：线性注意力的第二种擦除方向
The Query Knows What to Forget: A Second Erase Direction for Linear Attention
浏览论文内容
中文总结 AI 辅助
该研究针对线性注意力长上下文下的存储项干扰问题，提出查询衍生擦除方向（QED），可提升检索性能并在S-NIAH-1上使可用上下文长度约翻倍。
中文摘要 AI 辅助
线性注意力维持固定大小的状态。在长上下文场景下，大量存储项共享该状态，它们之间的干扰会降低检索性能。与此前所有增量规则模型一样，门控DeltaNet-2（GDN-2）从当前token的键（key）推导其擦除向量。然而，其读取过程中的干扰是通过查询（query）衡量的，而擦除步骤无法触及该干扰。我们提出查询衍生擦除方向（QED），QED添加了一种由查询推导且与键正交的第二种擦除方向。在快速权重视角下，键导向的增量编辑无法改变读取过程中与键正交的部分，它利用可编辑部分抵消沿查询衡量的旧状态内容。此外，QED在超过训练窗口的所有长度下均提升了检索性能，在S-NIAH-1数据集上将可用上下文长度提升了约一倍。
英文摘要
Linear attention keeps a state of fixed size. At long context, many stored items share this state, and interference between them degrades retrieval. Gated DeltaNet-2 (GDN-2), like every delta-rule model before it, derives its erase vector from the key of the current token. However, the interference in its reads is measured through the query, and the erase step cannot reach it. We introduce the Query-derived Erase Direction (QED). QED adds a second erase direction derived from the query and orthogonal to the key. In the fast-weight view, a key-directed delta edit cannot change the key-orthogonal part of a read. It uses the editable part to cancel old-state content measured along the query. It also improves retrieval at every length past the training window, and it about doubles the usable context length on S-NIAH-1.
