arXiv 2407.148142024-07-20cs.LG

FMamba：基于快速注意力机制的Mamba多元时间序列预测

FMamba: Mamba based on Fast-attention for Multivariate Time-series Forecasting

Institute of Advanced Technology, USTC（中国科学技术大学先进技术研究院）；Department of Automation, USTC（中国科学技术大学自动化系）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）

Shusen Ma, Yu Kang, Peng Bai, Yun-Bo Zhao

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对多元时间序列预测中Mamba模型无法关注变量间关系的问题，提出结合fast-attention与Mamba的FMamba框架，在保持低计算开销的同时实现了最先进的预测性能。

摘要：

在多元时间序列预测（MTSF）中，提取输入序列的时间相关性至关重要。虽然流行的基于Transformer的预测模型表现良好，但其二次计算复杂度导致了低效和高开销。最近出现的Mamba是一种选择性状态空间模型，凭借其强大的时间特征提取能力和线性计算复杂度，在许多领域展现出可喜的结果。然而，由于Mamba的单向性质，基于Mamba的通道独立预测模型无法像基于Transformer的模型那样关注所有变量之间的关系。为了解决这个问题，我们将fast-attention与Mamba相结合，提出了一个名为FMamba的MTSF新框架。在技术上，我们首先通过嵌入层提取输入变量的时间特征，然后通过fast-attention模块计算输入变量之间的依赖关系。随后，我们使用Mamba选择性地处理输入特征，并通过多层感知机块（MLP-block）进一步提取变量的时间依赖性。最后，FMamba通过投影器（一个线性层）获得预测结果。在八个公开数据集上的实验结果表明，FMamba能够在保持低计算开销的同时实现最先进的性能。

英文摘要：

In multivariate time-series forecasting (MTSF), extracting the temporal correlations of the input sequences is crucial. While popular Transformer-based predictive models can perform well, their quadratic computational complexity results in inefficiency and high overhead. The recently emerged Mamba, a selective state space model, has shown promising results in many fields due to its strong temporal feature extraction capabilities and linear computational complexity. However, due to the unilateral nature of Mamba, channel-independent predictive models based on Mamba cannot attend to the relationships among all variables in the manner of Transformer-based models. To address this issue, we combine fast-attention with Mamba to introduce a novel framework named FMamba for MTSF. Technically, we first extract the temporal features of the input variables through an embedding layer, then compute the dependencies among input variables via the fast-attention module. Subsequently, we use Mamba to selectively deal with the input features and further extract the temporal dependencies of the variables through the multi-layer perceptron block (MLP-block). Finally, FMamba obtains the predictive results through the projector, a linear layer. Experimental results on eight public datasets demonstrate that FMamba can achieve state-of-the-art performance while maintaining low computational overhead.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。