arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2302.00861cs.LG
收藏

SimMTM:一个用于掩码时间序列建模的简单预训练框架

SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling

Tsinghua University(清华大学)

Jiaxiang Dong, Haixu Wu, Haoran Zhang, Li Zhang, Jianmin Wang, Mingsheng Long

AI总结:

针对随机掩码破坏时间序列变化导致重建困难的问题,提出 SimMTM 预训练框架,通过流形外多邻居加权聚合恢复掩码点并学习流形局部结构,在预测和分类任务中取得最优微调性能。

摘要:

时间序列分析被广泛应用于众多领域。近年来,为了降低标注开销并使各种任务受益,自监督预训练引起了极大的关注。一个主流范式是掩码建模,它通过学习基于未掩码部分重建掩码内容,成功预训练了深度模型。然而,由于时间序列的语义信息主要包含在时间变化中,随机掩码部分时间点的标准方法会严重破坏时间序列的重要时间变化,使重建任务过于困难而无法指导表示学习。因此,我们提出了 SimMTM,一个用于掩码时间序列建模的简单预训练框架。通过将掩码建模与流形学习相关联,SimMTM 提出通过流形外多个邻居的加权聚合来恢复掩码时间点,通过从多个掩码序列中汇集被破坏但互补的时间变化来缓解重建任务。SimMTM 进一步学习揭示流形的局部结构,这有助于掩码建模。实验上,在两个典型的时间序列分析任务:预测和分类中,与最先进的时间序列预训练方法相比,SimMTM 实现了最先进的微调性能,涵盖了域内和跨域设置。

英文摘要:

Time series analysis is widely used in extensive areas. Recently, to reduce labeling expenses and benefit various tasks, self-supervised pre-training has attracted immense interest. One mainstream paradigm is masked modeling, which successfully pre-trains deep models by learning to reconstruct the masked content based on the unmasked part. However, since the semantic information of time series is mainly contained in temporal variations, the standard way of randomly masking a portion of time points will seriously ruin vital temporal variations of time series, making the reconstruction task too difficult to guide representation learning. We thus present SimMTM, a Simple pre-training framework for Masked Time-series Modeling. By relating masked modeling to manifold learning, SimMTM proposes to recover masked time points by the weighted aggregation of multiple neighbors outside the manifold, which eases the reconstruction task by assembling ruined but complementary temporal variations from multiple masked series. SimMTM further learns to uncover the local structure of the manifold, which is helpful for masked modeling. Experimentally, SimMTM achieves state-of-the-art fine-tuning performance compared to the most advanced time series pre-training methods in two canonical time series analysis tasks: forecasting and classification, covering both in- and cross-domain settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑