多头注意力：协作而不是串联
Multi-Head Attention: Collaborate Instead of Concatenate
摘要：
注意层广泛应用于自然语言处理（NLP）中，并开始影响计算机视觉架构。训练非常大的Transformer模型可以在这两个领域都有显着的改进，但是一旦训练好，这些网络就会显示出过度参数化的症状。例如，已知可以在不影响准确性的情况下修剪许多注意力头部。这项工作旨在加强目前对多个头如何相互作用的理解。出于观察，注意头学习冗余的关键字/查询投影，我们提出了一个合作的多头注意力层，使头部学习共享的投影。我们的方案减少了注意层中的参数数量，可以用作任何Transformer架构中的插入式替代。我们的实验证实，共享密钥/查询维度可以在语言理解，机器翻译和视觉中利用。我们还表明，它是可能的重新参数化一个预先训练的多头注意力层到我们的协作注意力层。协作多头注意力将关键字和查询投影的大小减少了4，以获得相同的精度和速度。代码是公开的。
英文摘要：
Attention layers are widely used in natural language processing (NLP) and are beginning to influence computer vision architectures. Training very large transformer models allowed significant improvement in both fields, but once trained, these networks show symptoms of over-parameterization. For instance, it is known that many attention heads can be pruned without impacting accuracy. This work aims to enhance current understanding on how multiple heads interact. Motivated by the observation that attention heads learn redundant key/query projections, we propose a collaborative multi-head attention layer that enables heads to learn shared projections. Our scheme decreases the number of parameters in an attention layer and can be used as a drop-in replacement in any transformer architecture. Our experiments confirm that sharing key/query dimensions can be exploited in language understanding, machine translation and vision. We also show that it is possible to re-parametrize a pre-trained multi-head attention layer into our collaborative attention layer. Collaborative multi-head attention reduces the size of the key and query projections by 4 for same accuracy and speed. Our code is public.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
