arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.20586cs.LGcs.AI
收藏

连续注意力(Continuum Attention)

Continuum Attention

University of Luxembourg(卢森堡大学)

Afshin Khadangi

AI总结:

该研究提出连续注意力(CA),用连续积分替代离散键值查找,通过解析推导梯度实现并行GPU计算,实验显示其在负对数似然上优于softmax和焦点注意力,是有前景的注意力替代方案。

摘要:

我们提出了连续注意力(Continuum Attention,CA),这是一种因果Transformer注意力算子,它用对连续分段线性记忆曲线的积分替代离散的键值查找。每个查询会沿该曲线诱导出指数相关性密度,对该密度与插值后的值进行积分,可从闭式指数矩(包括半 token 边界贡献)中得到精确归一化注意力。梯度可通过至多二阶矩解析推导得出,这使其能实现并行GPU实现,同时保留二次序列复杂度与标准Transformer解码器。与普通softmax和焦点注意力相比,CA分别将平均验证负对数似然降低了0.0200和0.0480 nat/字节,在所有配对比较中均胜出。另一组不相交验证切片显示,其相比最强的选定softmax基线具有0.0457 nat/字节的优势,且有配对95%置信区间。这些发现表明连续积分是一种有前景的注意力替代方案,同时也为更大规模的泛化留下了探索空间。

英文摘要:

Conventional causal attention aggregates values at discrete memory positions. We introduce Continuum Attention (CA), in which neighboring key and value vectors define a continuous, piecewise linear memory trajectory and each query retrieves information by normalized integration over its visible prefix. The segment integrals have closed forms in exponential moments, yielding analytically differentiable attention without numerical quadrature. The resulting operator preserves causal masking, normalized nonnegative effective weights, and the standard Transformer decoder structure. Controlled byte level experiments favor exact continuous integration over matched softmax attention. We additionally evaluate data efficiency, out of domain language modeling, zero-shot multiple choice prediction, robustness, and multilingual generalization. In multilingual evaluation, CA consistently outperforms all baselines across every evaluated language, achieving an average 5.8% reduction in bits per byte (BPB) relative to the strongest baseline.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑