arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2205.07100cs.CLcs.AIcs.MMcs.SDeess.AS
收藏

Multiformer:一种基于头部可配置transformer的直接语音翻译模型

Multiformer: A Head-Configurable Transformer-Based Model for Direct Speech Translation

TALP Research Center(TALP研究中心);Universitat Politècnica de Catalunya(加泰罗尼亚理工大学)

Gerard Sant, Gerard I. Gállego, Belen Alastruey, Marta R. Costa-Jussà

摘要:

基于transformer的模型已经在自然语言处理的多个领域取得了最先进的成果。然而,它的直接应用到语音任务是不平凡的。这种序列的性质带来了诸如长序列长度和相邻令牌之间的冗余之类的问题。因此,我们认为常规的自我注意机制可能并不适合于它,人们提出了不同的方法来克服这些问题,例如使用有效的注意力机制。然而,这些方法的使用通常伴随着成本,这是由信息丢失引起的性能降低。在这项研究中,我们提出了Multiformer,一个基于transformer的模型,它允许在每个头上使用不同的注意力机制。通过这样做,该模型能够将自我注意力偏向于提取更多样化的令牌交互,并减少信息损失。最后,我们执行的头的贡献的分析,我们观察到,这些架构中的所有头的相关性是均匀分布的获得更好的结果。我们的研究结果表明,沿着不同的头部和层混合注意力模式比我们的基线高出0.7 BLEU。

英文摘要:

Transformer-based models have been achieving state-of-the-art results in several fields of Natural Language Processing. However, its direct application to speech tasks is not trivial. The nature of this sequences carries problems such as long sequence lengths and redundancy between adjacent tokens. Therefore, we believe that regular self-attention mechanism might not be well suited for it. Different approaches have been proposed to overcome these problems, such as the use of efficient attention mechanisms. However, the use of these methods usually comes with a cost, which is a performance reduction caused by information loss. In this study, we present the Multiformer, a Transformer-based model which allows the use of different attention mechanisms on each head. By doing this, the model is able to bias the self-attention towards the extraction of more diverse token interactions, and the information loss is reduced. Finally, we perform an analysis of the head contributions, and we observe that those architectures where all heads relevance is uniformly distributed obtain better results. Our results show that mixing attention patterns along the different heads and layers outperforms our baseline by up to 0.7 BLEU.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑