arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2106.10840cs.CLcs.AI
收藏

更好地关注注意力:多语言多领域序列建模中的头部选择

Pay Better Attention to Attention: Head Selection in Multilingual and Multi-Domain Sequence Modeling

Facebook AI Research(脸书人工智能研究院)

Hongyu Gong, Yun Tang, Juan Pino, Xian Li

摘要:

多头注意力使每个注意力头从输入序列的不同部分收集显著信息,使其成为序列建模的强大机制。多语言和多领域学习是序列建模的常见场景,其中的关键挑战是最大限度地提高语言和领域之间的正迁移并减轻负迁移。在本文中,我们发现,非选择性的注意力共享是次优的实现良好的泛化在所有的语言和领域。我们进一步提出了注意力共享策略,以促进多语言和多域序列建模中的参数共享和专业化。我们的方法自动学习不同语言和领域的共享和专门注意力,以减轻它们的干扰。在各种任务,包括语音识别,文本到文本和语音到文本的翻译,所提出的注意力共享策略始终带来收益的序列模型建立在多头注意力。对于语音到文本的翻译,我们的方法产生的平均$+2.0$ BLEU的$13$语言方向在多语言设置和$+2.0$ BLEU的$3$域在多域设置。

英文摘要:

Multi-head attention has each of the attention heads collect salient information from different parts of an input sequence, making it a powerful mechanism for sequence modeling. Multilingual and multi-domain learning are common scenarios for sequence modeling, where the key challenge is to maximize positive transfer and mitigate negative transfer across languages and domains. In this paper, we find that non-selective attention sharing is sub-optimal for achieving good generalization across all languages and domains. We further propose attention sharing strategies to facilitate parameter sharing and specialization in multilingual and multi-domain sequence modeling. Our approach automatically learns shared and specialized attention heads for different languages and domains to mitigate their interference. Evaluated in various tasks including speech recognition, text-to-text and speech-to-text translation, the proposed attention sharing strategies consistently bring gains to sequence models built upon multi-head attention. For speech-to-text translation, our approach yields an average of $+2.0$ BLEU over $13$ language directions in multilingual setting and $+2.0$ BLEU over $3$ domains in multi-domain setting.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑