arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.16516cs.LG
收藏

用于时间序列预测的生成式预训练分层 Transformer

Generative Pretrained Hierarchical Transformer for Time Series Forecasting

University of Science and Technology of China(中国科学技术大学);Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院)

Zhiding Liu, Jiqian Yang, Mingyue Cheng, Yucong Luo, Zhi Li

AI总结:

本文提出 GPHT,通过多场景混合数据预训练和自回归分层 Transformer,实现单一模型在任意预测时长下的泛化预测,并在八个数据集上显著优于基线。

摘要:

近来,研究人员致力于通过引入先进的网络架构和自监督预训练策略来提高时间序列预测的准确性。尽管如此,现有方法仍存在两个关键缺陷。首先,这些方法通常依赖单一数据集进行训练,由于训练数据规模有限,模型的泛化能力受到限制。其次,单步生成模式被广泛采用,这需要定制化的预测头,忽视了输出序列中的时间依赖关系,并且在不同预测时长设置下还会导致训练成本增加。为解决这些问题,我们提出了一种新颖的用于预测的生成式预训练分层 transformer 架构,命名为 GPHT。GPHT 有两方面的关键设计。一方面,我们主张在通道独立假设下构建混合数据集来预训练模型,该数据集包含来自不同数据场景的多种数据集。这种方法显著扩大了训练数据规模,使模型能够发现时间序列数据中的共性,并促进更好地迁移到特定数据集。另一方面,GPHT 采用自回归预测方法,有效建模输出序列中的时间依赖关系。重要的是,该方法不需要定制化预测头,使单个模型能够在任意预测时长设置下进行预测。我们在八个数据集上与主流自监督预训练模型和监督模型进行了充分实验。结果表明,在传统长期预测任务中,GPHT 在各种微调以及零样本/少样本学习设置下均优于基线模型。我们公开了代码:https://github.com/icantnamemyself/GPHT。

英文摘要:

Recent efforts have been dedicated to enhancing time series forecasting accuracy by introducing advanced network architectures and self-supervised pretraining strategies. Nevertheless, existing approaches still exhibit two critical drawbacks. Firstly, these methods often rely on a single dataset for training, limiting the model's generalizability due to the restricted scale of the training data. Secondly, the one-step generation schema is widely followed, which necessitates a customized forecasting head and overlooks the temporal dependencies in the output series, and also leads to increased training costs under different horizon length settings. To address these issues, we propose a novel generative pretrained hierarchical transformer architecture for forecasting, named \textbf{GPHT}. There are two aspects of key designs in GPHT. On the one hand, we advocate for constructing a mixed dataset under the channel-independent assumption for pretraining our model, comprising various datasets from diverse data scenarios. This approach significantly expands the scale of training data, allowing our model to uncover commonalities in time series data and facilitating improved transfer to specific datasets. On the other hand, GPHT employs an auto-regressive forecasting approach, effectively modeling temporal dependencies in the output series. Importantly, no customized forecasting head is required, enabling \textit{a single model to forecast at arbitrary horizon settings.} We conduct sufficient experiments on eight datasets with mainstream self-supervised pretraining models and supervised models. The results demonstrated that GPHT surpasses the baseline models across various fine-tuning and zero/few-shot learning settings in the traditional long-term forecasting task. We make our codes publicly available\footnote{https://github.com/icantnamemyself/GPHT}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑