arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.05996cs.LGstat.ML
收藏

正交自注意力

Orthogonal Self-Attention

University of Oxford(牛津大学)

Leo Zhang, James Martens

AI总结:

本文提出正交自注意力机制,通过正交化注意力矩阵解决SSA的稳定性问题,使无跳过连接和归一化层的Transformer更易训练。

摘要:

Softmax 自注意力(SSA)是Transformer架构中的关键组件。然而,当在无跳过连接的架构中使用时,这些架构旨在提高表示学习,最近的研究指出由于诱导秩坍缩和不好的条件雅可比矩阵,SSA存在内在的不稳定性。在本工作中,我们设计了一种新的注意力机制:正交自注意力(OSA),旨在绕过SSA的问题,从而允许无跳过连接和归一化层的非因果Transformer更容易训练。特别是,OSA通过将查询-键值形成的斜对称矩阵通过矩阵指数映射,参数化注意力矩阵为正交。我们证明这可以实际实现,通过利用查询-键值的低秩结构,导致OSA的计算复杂度和内存成本与序列长度线性增长。此外,我们推导出一种初始化方案,我们证明该方案确保OSA的雅可比矩阵是良好条件的。

英文摘要:

Softmax Self-Attention (SSA) is a key component of Transformer architectures. However, when utilised within skipless architectures, which aim to improve representation learning, recent work has highlighted the inherent instability of SSA due to inducing rank collapse and poorly-conditioned Jacobians. In this work, we design a novel attention mechanism: Orthogonal Self-Attention (OSA), which aims to bypass these issues with SSA, in order to allow for (non-causal) Transformers without skip connections and normalisation layers to be more easily trained. In particular, OSA parametrises the attention matrix to be orthogonal via mapping a skew-symmetric matrix, formed from query-key values, through the matrix exponential. We show that this can be practically implemented, by exploiting the low-rank structure of our query-key values, resulting in the computational complexity and memory cost of OSA scaling linearly with sequence length. Furthermore, we derive an initialisation scheme for which we prove ensures that the Jacobian of OSA is well-conditioned.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑