时间复杂度为O(n log n)的序列处理算法
Neural Shuffle-Exchange Networks -- Sequence Processing in O(n log n) Time
摘要:
序列到序列处理中的一个关键要求是长范围依赖关系的建模。为此,绝大多数的国家的最先进的模型使用的注意力机制,这是O($n^2$)的复杂性,导致执行缓慢的长序列。我们引入了一个新的Shuffle-Exchange神经网络模型,用于具有O(log n)深度和O(nlog n)总复杂度的序列到序列任务。我们表明,这个模型是强大的,足以推断出有效的算法,常见的算法基准,包括排序,加法和乘法。我们评估我们的架构上具有挑战性的LAMBADA问答数据集,并将其与最先进的模型,使用注意力。我们的模型达到了有竞争力的准确性,并扩展到超过十万个元素的序列。我们相信,该模型有潜力建立更有效的架构,处理大型相关数据的语言建模,音乐生成和其他应用领域。
英文摘要:
A key requirement in sequence to sequence processing is the modeling of long range dependencies. To this end, a vast majority of the state-of-the-art models use attention mechanism which is of O($n^2$) complexity that leads to slow execution for long sequences. We introduce a new Shuffle-Exchange neural network model for sequence to sequence tasks which have O(log n) depth and O(n log n) total complexity. We show that this model is powerful enough to infer efficient algorithms for common algorithmic benchmarks including sorting, addition and multiplication. We evaluate our architecture on the challenging LAMBADA question answering dataset and compare it with the state-of-the-art models which use attention. Our model achieves competitive accuracy and scales to sequences with more than a hundred thousand of elements. We are confident that the proposed model has the potential for building more efficient architectures for processing large interrelated data in language modeling, music generation and other application domains.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。