长期序列预测是否需要复杂注意力机制和额外长输入？
Does Long-Term Series Forecasting Need Complex Attention and Extra Long Inputs?
AI总结：
本文提出轻量级周期注意力机制Periodformer和基于贝叶斯优化的多GPU异步并行算法MABO，证明长期序列预测无需复杂注意力与额外长输入，预测误差降低13%至26%，搜索时间减少46%。
摘要：
随着基于Transformer的模型在各种时间序列任务上取得了令人瞩目的性能，长期序列预测（LTSF）任务近年来也受到了广泛关注。然而，由于基于Transformer的方法固有的计算复杂性和对长序列的需求，其在LTSF任务上的应用仍存在两个需要进一步研究的主要问题：1）这些方法设计的稀疏注意力机制是否真正减少了在真实设备上的运行时间；2）这些模型是否需要额外的长输入序列来保证其性能？本文给出的答案是否定的。因此，为了更好地应对这两个问题，我们设计了一种轻量级的周期注意力机制（Periodformer），通过显式周期性聚合长期子序列，并利用内置邻近性聚合短期子序列，从而革新了子序列的聚合方式。同时，Periodformer中嵌入了一个门控机制，以调节注意力模块对预测结果的影响。此外，为了充分利用GPU进行快速超参数优化（例如，寻找合适的输入长度），我们提出了一种基于贝叶斯优化的多GPU异步并行算法（MABO）。MABO通过队列机制为每个GPU分配一个进程，然后一次创建多个试验进行异步并行搜索，这大大减少了搜索时间。与最先进的方法相比，Periodformer在多变量和单变量预测中的预测误差分别降低了13%和26%。此外，MABO在找到更好超参数的同时，将平均搜索时间减少了46%。总之，本文表明LTSF可能不需要复杂的注意力和额外的长输入序列。代码已在Github上开源。
英文摘要：
As Transformer-based models have achieved impressive performance on various time series tasks, Long-Term Series Forecasting (LTSF) tasks have also received extensive attention in recent years. However, due to the inherent computational complexity and long sequences demanding of Transformer-based methods, its application on LTSF tasks still has two major issues that need to be further investigated: 1) Whether the sparse attention mechanism designed by these methods actually reduce the running time on real devices; 2) Whether these models need extra long input sequences to guarantee their performance? The answers given in this paper are negative. Therefore, to better copy with these two issues, we design a lightweight Period-Attention mechanism (Periodformer), which renovates the aggregation of long-term subseries via explicit periodicity and short-term subseries via built-in proximity. Meanwhile, a gating mechanism is embedded into Periodformer to regulate the influence of the attention module on the prediction results. Furthermore, to take full advantage of GPUs for fast hyperparameter optimization (e.g., finding the suitable input length), a Multi-GPU Asynchronous parallel algorithm based on Bayesian Optimization (MABO) is presented. MABO allocates a process to each GPU via a queue mechanism, and then creates multiple trials at a time for asynchronous parallel search, which greatly reduces the search time. Compared with the state-of-the-art methods, the prediction error of Periodformer reduced by 13% and 26% for multivariate and univariate forecasting, respectively. In addition, MABO reduces the average search time by 46% while finding better hyperparameters. As a conclusion, this paper indicates that LTSF may not need complex attention and extra long input sequences. The code has been open sourced on Github.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
