用于BERT语言模型高效训练的对称点积注意力机制
Symmetric Dot-Product Attention for Efficient Training of BERT Language Models
AI总结：
本研究针对Transformer模型训练资源消耗大的问题，提出带成对系数的对称点积注意力机制，在类BERT预训练中提升了GLUE基准得分，同时减少参数量并加快收敛。
摘要：
Transformer架构最初作为机器翻译模型提出，如今已成为现代深度学习架构的基础，应用范围涵盖计算机视觉、自然语言处理等众多领域。如今，为应对日益复杂的任务，基于Transformer的模型规模被扩展至极巨量，需要越来越大的训练数据集，以及难以持续供给的计算资源。Transformer及其核心组件注意力机制的广泛应用，使其成为效率优化研究的核心目标。在本研究中，我们为Transformer架构提出的自注意力机制设计了一种替代兼容性函数。该函数利用传统缩放点积注意力学习表征中的重叠特性，构建出带成对系数的对称点积注意力。将这种新型对称注意力机制应用于类BERT模型的预训练时，其在GLUE基准上取得79.36的得分，而传统实现得分为78.74；同时可训练参数数量减少6%，收敛所需的训练步数减半。
英文摘要：
Initially introduced as a machine translation model, the Transformer architecture has now become the foundation for modern deep learning architecture, with applications in a wide range of fields, from computer vision to natural language processing. Nowadays, to tackle increasingly more complex tasks, Transformer-based models are stretched to enormous sizes, requiring increasingly larger training datasets, and unsustainable amount of compute resources. The ubiquitous nature of the Transformer and its core component, the attention mechanism, are thus prime targets for efficiency research. In this work, we propose an alternative compatibility function for the self-attention mechanism introduced by the Transformer architecture. This compatibility function exploits an overlap in the learned representation of the traditional scaled dot-product attention, leading to a symmetric with pairwise coefficient dot-product attention. When applied to the pre-training of BERT-like models, this new symmetric attention mechanism reaches a score of 79.36 on the GLUE benchmark against 78.74 for the traditional implementation, leads to a reduction of 6% in the number of trainable parameters, and reduces the number of training steps required before convergence by half.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
