利用扩张自我注意捕获多分辨率上下文
Capturing Multi-Resolution Context by Dilated Self-Attention
摘要：
自注意力已经成为一种重要且广泛使用的神经网络组件，有助于为各种应用建立新的最先进的结果，例如机器翻译和自动语音识别（ASR）。然而，自注意的计算复杂度与输入序列长度成二次方增长。这对于诸如ASR之类的应用可能是特别有问题的，其中从话语生成的输入序列可能相对较长。在这项工作中，我们提出了一个限制性的自我注意力和膨胀机制，我们称之为膨胀的自我注意力的组合。受限的自关注允许以高分辨率关注查询的相邻帧，并且膨胀机制总结远距离信息以允许以较低分辨率关注它。不同的方法来总结遥远的帧进行了研究，如子采样，均值池，和基于注意力池。ASR结果表明，与单独的限制性自我注意相比，有了很大的改进，与基于全序列的自我注意相比，实现了类似的结果，而计算成本只有一小部分。
英文摘要：
Self-attention has become an important and widely used neural network component that helped to establish new state-of-the-art results for various applications, such as machine translation and automatic speech recognition (ASR). However, the computational complexity of self-attention grows quadratically with the input sequence length. This can be particularly problematic for applications such as ASR, where an input sequence generated from an utterance can be relatively long. In this work, we propose a combination of restricted self-attention and a dilation mechanism, which we refer to as dilated self-attention. The restricted self-attention allows attention to neighboring frames of the query at a high resolution, and the dilation mechanism summarizes distant information to allow attending to it with a lower resolution. Different methods for summarizing distant frames are studied, such as subsampling, mean-pooling, and attention-based pooling. ASR results demonstrate substantial improvements compared to restricted self-attention alone, achieving similar results compared to full-sequence based self-attention with a fraction of the computational costs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
