arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2004.04662cs.LGcs.SDeess.AS
收藏

用于长序列快速处理的残差混洗交换网络

Residual Shuffle-Exchange Networks for Fast Processing of Long Sequences

Institute of Mathematics and Computer Science, University of Latvia(拉脱维亚大学数学与计算机科学研究所)

Andis Draguns, Emīls Ozoliņš, Agris Šostaks, Matīss Apinis, Kārlis Freivalds

摘要:

注意力是序列处理中常用的一种机制,但它的复杂度为O(n^2),这使得它不能应用于长序列。最近引入的神经洗牌交换网络提供了一个计算效率高的替代方案,使长期依赖关系的建模在O(n log n)时间。然而,该模型是相当复杂的,涉及到一个复杂的门控机制,从门控经常性单位。在本文中,我们提出了一个简单的和轻量级的变体的洗牌交换网络,这是基于一个残差网络采用GELU和层归一化。所提出的架构不仅可以扩展到更长的序列,而且收敛速度更快,精度更高。它在LAMBADA语言建模任务上超越了Shuffle-Exchange网络,并在MusicNet数据集上实现了最先进的音乐转录性能,同时在参数数量上也很高效。我们展示了如何将改进的Shuffle-Exchange网络与卷积层相结合,将其作为长序列处理应用中有用的构建块。

英文摘要:

Attention is a commonly used mechanism in sequence processing, but it is of O(n^2) complexity which prevents its application to long sequences. The recently introduced neural Shuffle-Exchange network offers a computation-efficient alternative, enabling the modelling of long-range dependencies in O(n log n) time. The model, however, is quite complex, involving a sophisticated gating mechanism derived from the Gated Recurrent Unit. In this paper, we present a simple and lightweight variant of the Shuffle-Exchange network, which is based on a residual network employing GELU and Layer Normalization. The proposed architecture not only scales to longer sequences but also converges faster and provides better accuracy. It surpasses the Shuffle-Exchange network on the LAMBADA language modelling task and achieves state-of-the-art performance on the MusicNet dataset for music transcription while being efficient in the number of parameters. We show how to combine the improved Shuffle-Exchange network with convolutional layers, establishing it as a useful building block in long sequence processing applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑