arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2204.09028cs.CLcs.SDeess.AS
收藏

论直接引语翻译中的注意局部性

On the Locality of Attention in Direct Speech Translation

TALP Research Center(TALP研究中心);Universitat Politècnica de Catalunya(加泰罗尼亚理工大学)

Belen Alastruey, Javier Ferrando, Gerard I. Gállego, Marta R. Costa-jussà

摘要:

Transformers在多个NLP任务中取得了最先进的结果。然而,自我注意力机制的复杂性与序列长度成二次关系,这对涉及长序列的任务(例如语音领域)造成了障碍。在本文中,我们讨论了自我注意的直接语音翻译的有用性。首先,我们分析了编码器自注意力中的逐层令牌贡献,揭示了局部对角模式。为了证明一些注意力权重是可以避免的,我们建议用局部有效的注意力来代替标准的自我注意力,并根据分析结果设置所使用的上下文量。通过这种方法,我们的模型匹配基线性能,并通过跳过标准注意力丢弃的权重的计算来提高效率。

英文摘要:

Transformers have achieved state-of-the-art results across multiple NLP tasks. However, the self-attention mechanism complexity scales quadratically with the sequence length, creating an obstacle for tasks involving long sequences, like in the speech domain. In this paper, we discuss the usefulness of self-attention for Direct Speech Translation. First, we analyze the layer-wise token contributions in the self-attention of the encoder, unveiling local diagonal patterns. To prove that some attention weights are avoidable, we propose to substitute the standard self-attention with a local efficient one, setting the amount of context used based on the results of the analysis. With this approach, our model matches the baseline performance, and improves the efficiency by skipping the computation of those weights that standard attention discards.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑