arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.01294cs.CLcs.LG
收藏

不要阅读一切:用于线性注意力的曲率条件查询

Don't Read Everything: A Curvature-Conditioned Query for Linear Attention

Nanyang Technological University(南洋理工大学);National University of Singapore(国立新加坡大学);VinUniversity(文理大学)

Dong Le, Thong Nguyen, Cong-Duy Nguyen, Anh Tuan Luu

AI总结:

针对线性注意力在上下文检索和长上下文任务中的不足,提出曲率条件查询(CCQ)机制,通过二阶泰勒展开构建局部二次模型,利用运行键协方差收缩查询向量,仅修改读取步骤,兼容现有线性注意力骨干,在困惑度、零样本下游准确率、检索和长上下文任务上取得提升。

摘要:

线性注意力通过维护一个循环的快速权重状态,降低了 softmax 注意力的二次成本,但在上下文检索和长上下文任务中始终落后。现有的补救措施通过门控、增量更新或核特征映射作用于记忆的写入侧,但读取步骤保持不变:每个过去的键对输出都有加性贡献,因此有用的目标被存储向量的大多数稀释。我们借用 softmax 几何的一个特定部分来构建一个廉价的读取时查询收缩。在等向注意力点处对 softmax 对数配分函数进行二阶泰勒展开,得到一个局部二次模型,其曲率与运行键协方差一致,该量可以通过与线性注意力状态相同的循环/分块机制来维护。相关的线性算子在查询读取状态之前,沿着记忆的高密度方向收缩查询。我们将这种机制称为曲率条件查询(CCQ)。CCQ 仅修改读取步骤,并且可以与任何线性注意力骨干组合。将其附加到 GLA 和 Gated DeltaNet 上,它在困惑度、零样本下游准确率、训练上下文内外的 S-NIAH 检索、从 4K 到 20K 的长度外推困惑度以及 LongBench 准确率上均有提升,且额外成本很小。

英文摘要:

Linear attention reduces the quadratic cost of softmax attention by maintaining a recurrent fast-weight state, but it consistently lags on in-context retrieval and long-context tasks. Existing remedies act on the write side of memory through gating, delta updates, or kernel feature maps, but the read step is left unchanged: every past key contributes additively to the output, so useful targets are diluted by the bulk of stored vectors. We borrow one specific piece of softmax's geometry to construct a cheap read-time contraction of the query. A second-order Taylor expansion of the softmax log-partition at the isotropic-attention point gives a local quadratic model whose curvature coincides with the running key covariance, a quantity that can be maintained with the same recurrent/chunkwise mechanism as the linear-attention state. The associated linear operator contracts the query along the high-variance directions of memory before it reads the state. We call this mechanism Curvature-Conditioned Query (CCQ). CCQ modifies only the read step and is composable with any linear-attention backbone. Attached to GLA and Gated DeltaNet, it improves perplexity, zero-shot downstream accuracy, S-NIAH retrieval at and beyond the training context, length-extrapolation perplexity from 4K to 20K, and LongBench accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑