arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.02279cs.LGcs.AIstat.ML
收藏

DRFormer:利用多样化感受野进行长时间序列预测的多尺度 Transformer

DRFormer: Multi-Scale Transformer Utilizing Diverse Receptive Fields for Long Time-Series Forecasting

East China Normal University(华东师范大学);Fudan University(复旦大学);Shanghai Jiao Tong University(上海交通大学)

Ruixin Ding, Yuqi Chen, Yu-Ting Lan, Wei Zhang

AI总结:

针对长期时间序列预测中固定 patch 难以捕捉多尺度特征的问题,本文提出 DRFormer,通过动态 tokenizer、稀疏学习、多尺度 Transformer 和组感知旋转位置编码提升预测性能。

摘要:

长期时间序列预测(LTSF)已被广泛应用于金融、交通预测及其他领域。近年来,基于 patch 的 Transformer 成为一种很有前景的方法,其将数据分割为子级 patch,并将其作为输入 token。然而,现有方法大多依赖预先设定的 patch 长度,这需要专家知识,并且在捕捉不同尺度上的多样化特征方面存在挑战。此外,时间序列数据在不同时间尺度上呈现出多样的变化和波动,而传统方法难以有效建模。本文提出一种带有动态稀疏学习算法的动态 tokenizer,用于捕捉时间序列数据的多样化感受野和稀疏模式。为构建层次化感受野,我们开发了一个多尺度 Transformer 模型,并结合多尺度序列提取,能够捕捉多分辨率特征。此外,我们引入组感知旋转位置编码技术,以增强不同时间尺度表示之间的组内和组间位置感知能力。所提出的模型名为 DRFormer,在多个真实世界数据集上进行了评估,实验结果证明其相较于现有方法具有优越性。代码可在 https://github.com/ruixindingECNU/DRFormer 获取。

英文摘要:

Long-term time series forecasting (LTSF) has been widely applied in finance, traffic prediction, and other domains. Recently, patch-based transformers have emerged as a promising approach, segmenting data into sub-level patches that serve as input tokens. However, existing methods mostly rely on predetermined patch lengths, necessitating expert knowledge and posing challenges in capturing diverse characteristics across various scales. Moreover, time series data exhibit diverse variations and fluctuations across different temporal scales, which traditional approaches struggle to model effectively. In this paper, we propose a dynamic tokenizer with a dynamic sparse learning algorithm to capture diverse receptive fields and sparse patterns of time series data. In order to build hierarchical receptive fields, we develop a multi-scale Transformer model, coupled with multi-scale sequence extraction, capable of capturing multi-resolution features. Additionally, we introduce a group-aware rotary position encoding technique to enhance intra- and inter-group position awareness among representations across different temporal scales. Our proposed model, named DRFormer, is evaluated on various real-world datasets, and experimental results demonstrate its superiority compared to existing methods. Our code is available at: https://github.com/ruixindingECNU/DRFormer.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑