arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.10410cs.LG
收藏

LUCID:带有预条件表示的注意力

LUCID: Attention with Preconditioned Representations

The University of Texas at Austin(德克萨斯大学奥斯汀分校);Google(谷歌公司)

Sai Surya Duvvuri, Nirmal Patel, Nilesh Gupta, Inderjit S. Dhillon

AI总结:

LUCID通过预条件表示改进注意力机制,提升长上下文检索任务性能,尤其在BABI Long和RULER任务中表现更优。

摘要:

基于softmax的点积注意力是Transformer架构的核心,使上下文学习等能力变得显著。然而,随着上下文长度的增加,softmax函数的根本限制显现出来:它倾向于将概率质量扩散到无关的token,从而在长序列场景中降低性能。此外,试图通过降低softmax温度来增强焦点会因梯度消失而影响可学习性。我们引入了LUCID注意力,这是一种对注意力概率应用预条件器的架构修改。该预条件器源自指数化的键-键相似性,在再生核希尔伯特空间中最小化键之间的重叠,从而允许查询在众多键中准确聚焦于重要的键,其计算复杂度与标准注意力相同。此外,LUCID基于预条件的检索方法避免了低温度和与之相关的可学习性问题。我们通过训练约10亿参数的语言模型,评估至128000个token。我们的结果表明,在长上下文检索任务中取得了显著提升,特别是在BABI Long、RULER、SCROLLS和LongBench的检索任务中。例如,LUCID在BABI Long任务中相比标准注意力提升了18%,在RULER多针性能中提升了14%。

英文摘要:

Softmax-based dot-product attention is a cornerstone of Transformer architectures, enabling remarkable capabilities such as in-context learning. However, as context lengths increase, a fundamental limitation of the softmax function emerges: it tends to diffuse probability mass to irrelevant tokens degrading performance in long-sequence scenarios. Furthermore, attempts to sharpen focus by lowering softmax temperature hinder learnability due to vanishing gradients. We introduce LUCID Attention, an architectural modification that applies a preconditioner to the attention probabilities. This preconditioner, derived from exponentiated key-key similarities, minimizes overlap between the keys in a Reproducing Kernel Hilbert Space, thus allowing the query to focus on important keys among large number of keys accurately with same computational complexity as standard attention. Additionally, LUCID's preconditioning-based approach to retrieval bypasses the need for low temperature and the learnability problems associated with it. We validate our approach by training ~1 billion parameter language models evaluated on up to 128K tokens. Our results demonstrate significant gains on long-context retrieval tasks, specifically retrieval tasks from BABILong, RULER, SCROLLS and LongBench. For instance, LUCID achieves up to 18% improvement in BABILong and 14% improvement in RULER multi-needle performance compared to standard attention.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑