arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.13033cs.LGcs.AIcs.CL
收藏

在注意力机制中扩展双向跨度和跨度违规

Scaling Bidirectional Spans and Span Violations in Attention Mechanism

Department of Physics, Sogang University, Seoul, Korea(物理系,ソガン大学,首尔,韩国);Department of Physics Education, Seoul National University, Seoul, Korea(物理教育系,首尔国立大学,首尔,韩国)

Jongwook Kim, Sangheon Yun, Sukjin Yoon

AI总结:

本文提出了一种优化框架,通过非对称投影分解注意力梯度,提升序列建模性能。

摘要:

传统的O(N²)Transformer仍然是序列建模的实证性能前沿,其训练可以通过解决几何效率低下问题进一步优化。我们提出了一种优化框架,利用非对称投影将反向传递梯度分解为平行跨度和正交违规,同时保持标准前向传递QKV结构不变。通过在各种分解和投影设置下一致的实验验证,我们提供了强有力的理论证据:标准注意力梯度是次优的。我们证明了选择性地扩展这些组件,主要集中在0阶双向并行跨度上,可以产生最有效的学习信号。在有限的WikiText-2数据集上,使用粗略配置,该方法实现了验证损失减少0.56%,证实了框架的基本有效性,并暗示在更大数据集和更深层次训练制度上可能获得显著收益。

英文摘要:

The canonical $O(N^2)$ Transformer remains the empirical performance frontier in sequence modeling, and its training can be further optimized by addressing geometric inefficiency. We propose an optimization framework that leverages an asymmetric projection to decompose the backward-pass gradients into parallel spans and orthogonal violations, while keeping the canonical forward-pass $QKV$ structure intact. Through consistent experimental validation across various decomposition and projection setups, we provide strong theoretical evidence: the standard attention gradient is suboptimal. We demonstrated that selectively scaling these components, focusing primarily on $0^{th}$ order bidirectional parallel spans, yields the most effective learning signal. On the limited WikiText-2 dataset, and using a crude configuration, this method achieved a $0.56\%$ reduction in validation loss, confirming the framework's fundamental validity and suggesting significant potential gains on larger datasets and deeper training regimes

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑