跳层注意力：弥合 Transformer 中的抽象依赖与细节依赖
Skip-Layer Attention: Bridging Abstract and Detailed Dependencies in Transformers
AI总结：
本文提出 Skip-Layer Attention（SLA），允许层查询同时与当前层和前一层的键值交互，以无额外计算开销的方式增强多头注意力多样性，并在语言建模任务中验证其性能优势。
摘要：
Transformer 架构通过有效处理长程依赖，显著推动了深度学习的发展，尤其是在自然语言处理领域。然而，随着对复杂关系理解需求的增长，改进 Transformer 的架构变得至关重要。本文提出 Skip-Layer Attention（SLA，跳层注意力），通过使非相邻层之间能够进行直接注意力交互来增强 Transformer 模型。该方法提升了模型捕获高层抽象特征与低层细节之间依赖关系的能力。通过促进这些不同特征层级之间的直接注意力，我们的方法克服了当前 Transformer 的局限——这些模型通常依赖于次优的层内注意力。我们的实现扩展了 Transformer 的功能，使给定层中的查询能够与来自当前层和前一层的键和值交互，从而在不增加计算负担的情况下增强多头注意力的多样性。大量实验表明，我们增强后的 Transformer 模型在语言建模任务中取得了更优性能，凸显了跳层注意力机制的有效性。
英文摘要：
The Transformer architecture has significantly advanced deep learning, particularly in natural language processing, by effectively managing long-range dependencies. However, as the demand for understanding complex relationships grows, refining the Transformer's architecture becomes critical. This paper introduces Skip-Layer Attention (SLA) to enhance Transformer models by enabling direct attention between non-adjacent layers. This method improves the model's ability to capture dependencies between high-level abstract features and low-level details. By facilitating direct attention between these diverse feature levels, our approach overcomes the limitations of current Transformers, which often rely on suboptimal intra-layer attention. Our implementation extends the Transformer's functionality by enabling queries in a given layer to interact with keys and values from both the current layer and one preceding layer, thus enhancing the diversity of multi-head attention without additional computational burden. Extensive experiments demonstrate that our enhanced Transformer model achieves superior performance in language modeling tasks, highlighting the effectiveness of our skip-layer attention mechanism.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
