arXiv 2410.226492024-11-21cs.LG

WaveRoRA：用于多元时间序列预测的小波旋转路由注意力

WaveRoRA: Wavelet Rotary Route Attention for Multivariate Time Series Forecasting

School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications（北京邮电大学计算机学院（国家示范性软件学院））；School of Computer Science and Engineering, University of Texas at Arlington（德克萨斯大学阿灵顿分校计算机科学与工程学院）

Aobo Liang, Yan Sun, Nadra Guizani

备注：: Model architecture changed

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对多元时间序列预测中时频特征分离和Softmax注意力二次复杂度问题，本文提出WaveRoRA框架，在小波域中利用旋转路由注意力RoRA以线性复杂度捕捉序列间依赖，在八个数据集上以更低计算成本超越现有最优模型。

摘要：

近年来，基于Transformer的模型（Transformers）在多元时间序列预测（MTSF）中取得了显著成功。然而，以往的工作侧重于从时域或频域提取特征，这不足以捕捉趋势和周期性特征。为了解决这一问题，我们提出了一种小波学习框架来建模时间序列数据的复杂时间依赖关系。小波域同时整合了时间和频率信息，允许在不同尺度上分析信号的局部特征。此外，Transformers使用的Softmax自注意力机制具有二次复杂度，在捕捉长期依赖时会导致过高的计算成本。因此，我们提出了一种新颖的注意力机制：旋转路由注意力（RoRA）。与Softmax注意力不同，RoRA利用旋转位置嵌入将相对位置信息注入序列token，并引入少量路由token $r$ 来聚合来自 $KV$ 矩阵的信息并将其重新分配到 $Q$ 矩阵，从而提供线性复杂度。我们进一步提出了WaveRoRA，它利用RoRA在小波域中捕捉序列间的依赖关系。我们在八个真实世界数据集上进行了广泛实验。结果表明，WaveRoRA在保持较低计算成本的同时，优于现有的最先进模型。我们的代码可在 https://github.com/Leopold2333/WaveRoRA 获取。

英文摘要：

In recent years, Transformer-based models (Transformers) have achieved significant success in multivariate time series forecasting (MTSF). However, previous works focus on extracting features either from the time domain or the frequency domain, which inadequately captures the trends and periodic characteristics. To address this issue, we propose a wavelet learning framework to model complex temporal dependencies of the time series data. The wavelet domain integrates both time and frequency information, allowing for the analysis of local characteristics of signals at different scales. Additionally, the Softmax self-attention mechanism used by Transformers has quadratic complexity, which leads to excessive computational costs when capturing long-term dependencies. Therefore, we propose a novel attention mechanism: Rotary Route Attention (RoRA). Unlike Softmax attention, RoRA utilizes rotary position embeddings to inject relative positional information to sequence tokens and introduces a small number of routing tokens $r$ to aggregate information from the $KV$ matrices and redistribute it to the $Q$ matrix, offering linear complexity. We further propose WaveRoRA, which leverages RoRA to capture inter-series dependencies in the wavelet domain. We conduct extensive experiments on eight real-world datasets. The results indicate that WaveRoRA outperforms existing state-of-the-art models while maintaining lower computational costs. Our code is available at https://github.com/Leopold2333/WaveRoRA.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。