arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.02368cs.LGstat.ML
收藏

Timer:生成式预训练Transformer是大型时间序列模型

Timer: Generative Pre-trained Transformers Are Large Time Series Models

BNRist, Tsinghua University(清华大学BNRist)

Yong Liu, Haoran Zhang, Chenyu Li, Xiangdong Huang, Jianmin Wang, Mingsheng Long

AI总结:

本文提出Timer,通过将异构时间序列统一为S3格式并进行GPT式生成预训练,构建大型时间序列模型,统一处理预测、插补和异常检测任务。

摘要:

深度学习对时间序列分析的发展做出了显著贡献。然而,深度模型在现实世界数据稀缺场景中可能遇到性能瓶颈,而这一瓶颈可能因当前基准上小模型的性能饱和而被掩盖。与此同时,大型模型通过大规模预训练在这些场景中展现出强大的能力。随着大型语言模型的出现,持续进展不断取得,展现出少样本泛化、可扩展性和任务通用性等前所未有的能力,而这些能力在小规模深度模型中并不具备。为了改变从头训练特定场景小模型的现状,本文旨在早期开发大型时间序列模型(LTSM)。在预训练阶段,我们构建了包含多达10亿个时间点的大规模数据集,将异构时间序列统一为单序列序列(S3)格式,并开发了面向LTSM的GPT风格架构。为了满足多样化的应用需求,我们将时间序列的预测、插补和异常检测转化为统一的生成式任务。本研究的成果是一个时间序列Transformer(Timer),它通过下一token预测进行生成式预训练,并适应各种下游任务,展现出作为LTSM的潜力。代码和数据集可在以下地址获取:https://github.com/thuml/Large-Time-Series-Model。

英文摘要:

Deep learning has contributed remarkably to the advancement of time series analysis. Still, deep models can encounter performance bottlenecks in real-world data-scarce scenarios, which can be concealed due to the performance saturation with small models on current benchmarks. Meanwhile, large models have demonstrated great powers in these scenarios through large-scale pre-training. Continuous progress has been achieved with the emergence of large language models, exhibiting unprecedented abilities such as few-shot generalization, scalability, and task generality, which are however absent in small deep models. To change the status quo of training scenario-specific small models from scratch, this paper aims at the early development of large time series models (LTSM). During pre-training, we curate large-scale datasets with up to 1 billion time points, unify heterogeneous time series into single-series sequence (S3) format, and develop the GPT-style architecture toward LTSMs. To meet diverse application needs, we convert forecasting, imputation, and anomaly detection of time series into a unified generative task. The outcome of this study is a Time Series Transformer (Timer), which is generative pre-trained by next token prediction and adapted to various downstream tasks with promising capabilities as an LTSM. Code and datasets are available at: https://github.com/thuml/Large-Time-Series-Model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑