arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2301.01772cs.LG
收藏

Infomaxformer:最大熵Transformer在长时间序列预测中的应用

Infomaxformer: Maximum Entropy Transformer for Long Time-Series Forecasting Problem

University of Science and Technology of China(中国科学技术大学);Jiaxing University(嘉兴学院)

Peiwang Tang, Xianchao Zhang

摘要:

Transformer架构在诸如自然语言处理(NLP)和计算机视觉(CV)等许多任务中产生了最先进的结果,因为能够有效地捕获输入序列之间的精确远程依赖性耦合。然而,这种先进的能力,二次的时间复杂度和高内存使用,防止Transformer处理长时间序列预测问题(LTFP)。为了解决这些困难:(1)重新审视了香草自我注意的学习注意模式,基于最大熵原理重新设计了自我注意的计算方法。(ii)本文提出了一种新的稀疏化自注意的方法,该方法可以避免由于随机抽样而丢失更重要的自注意分数。(iii)我们提出了Keys/Values Distilling方法,其动机是原始自注意图中的大量特征是冗余的,这可以进一步降低时间和空间复杂度,并使输入更长的时间序列成为可能。最后,我们提出了一种将编码器-解码器架构与季节趋势分解相结合的方法,即,使用编码器-解码器架构来捕获更具体的季节部分。在多个大规模数据集上的大量实验表明,本文提出的方法明显优于现有方法。我们希望这为Transformer解决LTFP开辟一个新的解决方案,并探索Transformer架构捕获更长时间依赖性的能力。

英文摘要:

The Transformer architecture yields state-of-the-art results in many tasks such as natural language processing (NLP) and computer vision (CV), since the ability to efficiently capture the precise long-range dependency coupling between input sequences. With this advanced capability, however, the quadratic time complexity and high memory usage prevents the Transformer from dealing with long time-series forecasting problem (LTFP). To address these difficulties: (i) we revisit the learned attention patterns of the vanilla self-attention, redesigned the calculation method of self-attention based the Maximum Entropy Principle. (ii) we propose a new method to sparse the self-attention, which can prevent the loss of more important self-attention scores due to random sampling.(iii) We propose Keys/Values Distilling method motivated that a large amount of feature in the original self-attention map is redundant, which can further reduce the time and spatial complexity and make it possible to input longer time-series. Finally, we propose a method that combines the encoder-decoder architecture with seasonal-trend decomposition, i.e., using the encoder-decoder architecture to capture more specific seasonal parts. A large number of experiments on several large-scale datasets show that our Infomaxformer is obviously superior to the existing methods. We expect this to open up a new solution for Transformer to solve LTFP, and exploring the ability of the Transformer architecture to capture much longer temporal dependencies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑