Alleviating the Inequality of Attention Heads for Neural Machine Translation
ByteDance AI Lab（字节跳动人工智能实验室）；Nanjing University（南京大学）；Peng Cheng Laboratory（鹏城实验室）
英文摘要：
Recent studies show that the attention heads in Transformer are not equal. We relate this phenomenon to the imbalance training of multi-head attention and the model dependence on specific heads. To tackle this problem, we propose a simple masking method: HeadMask, in two specific ways. Experiments show that translation improvements are achieved on multiple language pairs. Subsequent empirical analyses also support our assumption and confirm the effectiveness of the method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
