什么时候自我注意力可以被前馈层取代？
When Can Self-Attention Be Replaced by Feed Forward Layers?
摘要：
最近，自注意力模型，如Transformers在语音识别中已经给出了与递归神经网络系统相比具有竞争力的结果。自我注意力模型表现出色的关键因素是它们能够捕捉时间关系，而不受两个相关事件之间距离的限制。然而，我们注意到，学习的上下文范围从较低的自我注意层到较高的自我注意层逐渐增加，而声学事件通常以从左到右的顺序在短时间内发生。这就引出了一个问题：对于语音识别，对于Transformers的编码器中的自注意上层，整个序列的全局视图仍然重要吗？为了研究这一点，我们用前馈层取代了这些自我关注层。在我们的语音识别实验（《华尔街日报》和Switchboard）中，我们确实观察到了一个有趣的结果：用前馈层取代编码器中的上层自注意层不会导致性能下降，甚至会有轻微的提高。我们的实验提供了自我注意层如何处理语音信号的见解，从而得出结论，编码器的较低的自我注意层编码足够广泛的输入，因此在上层学习进一步的上下文信息是不必要的。
英文摘要：
Recently, self-attention models such as Transformers have given competitive results compared to recurrent neural network systems in speech recognition. The key factor for the outstanding performance of self-attention models is their ability to capture temporal relationships without being limited by the distance between two related events. However, we note that the range of the learned context progressively increases from the lower to upper self-attention layers, whilst acoustic events often happen within short time spans in a left-to-right order. This leads to a question: for speech recognition, is a global view of the entire sequence still important for the upper self-attention layers in the encoder of Transformers? To investigate this, we replace these self-attention layers with feed forward layers. In our speech recognition experiments (Wall Street Journal and Switchboard), we indeed observe an interesting result: replacing the upper self-attention layers in the encoder with feed forward layers leads to no performance drop, and even minor gains. Our experiments offer insights to how self-attention layers process the speech signal, leading to the conclusion that the lower self-attention layers of the encoder encode a sufficiently wide range of inputs, hence learning further contextual information in the upper layers is unnecessary.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
