arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.05370cs.LGcs.AI
收藏

注意力作为时间序列预测的稳健表示

Attention as Robust Representation for Time Series Forecasting

Alibaba Group(阿里巴巴集团);Meta Platforms, Inc(元平台公司)

PeiSong Niu, Tian Zhou, Xue Wang, Liang Sun, Rong Jin

AI总结:

本研究提出将注意力权重作为时间序列的主要表示,利用全局地标和局部窗口构建稳健核表示,在不改变核心架构的情况下将多变量预测MSE降低3.6%,并可作为通用组件替换现有分块嵌入方案。

摘要:

时间序列预测对许多实际应用至关重要,基于Transformer的模型因其在自然语言处理和计算机视觉中的出色表现而日益普及。Transformer的关键特性——注意力机制,通过动态融合嵌入来增强数据表示,通常将注意力权重视为副产品角色。然而,以噪声和非平稳性为特征的时间序列数据给预测带来了重大挑战。我们的方法将注意力权重提升为时间序列的主要表示,利用数据点之间的时间关系来提高预测准确性。我们的研究表明,使用全局地标和局部窗口构建的注意力图可作为数据点的稳健核表示,能够抵御分布中的噪声和偏移。我们的方法优于最先进的模型,在不改变核心神经网络架构的情况下,将多变量时间序列预测中的均方误差(MSE)显著降低了3.6%。它作为一种通用组件,可轻松替换基于Transformer模型中的近期基于分块的嵌入方案,从而提升其性能。

英文摘要:

Time series forecasting is essential for many practical applications, with the adoption of transformer-based models on the rise due to their impressive performance in NLP and CV. Transformers' key feature, the attention mechanism, dynamically fusing embeddings to enhance data representation, often relegating attention weights to a byproduct role. Yet, time series data, characterized by noise and non-stationarity, poses significant forecasting challenges. Our approach elevates attention weights as the primary representation for time series, capitalizing on the temporal relationships among data points to improve forecasting accuracy. Our study shows that an attention map, structured using global landmarks and local windows, acts as a robust kernel representation for data points, withstanding noise and shifts in distribution. Our method outperforms state-of-the-art models, reducing mean squared error (MSE) in multivariate time series forecasting by a notable 3.6% without altering the core neural network architecture. It serves as a versatile component that can readily replace recent patching based embedding schemes in transformer-based models, boosting their performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑