arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.12372cs.LG
收藏

跨域预训练语言模型用于可迁移时间序列表示

Cross-Domain Pre-training with Language Models for Transferable Time Series Representations

University of Science and Technology of China(中国科学技术大学);State Key Laboratory of Cognitive Intelligence(认知智能国家重点实验室)

Mingyue Cheng, Xiaoyu Tao, Qi Liu, Hao Zhang, Yiheng Chen, Defu Lian

AI总结:

提出CrossTimeNet框架,通过时间序列分词与高比例损坏令牌预测,利用预训练语言模型初始化,实现跨域自监督预训练,提升下游时间序列分类性能。

摘要:

自监督预训练(SSL)的进展显著推动了可迁移时间序列表示学习领域的发展,这对于增强下游任务非常有用。尽管有效,但大多数现有工作难以实现跨域SSL预训练,错失了整合不同领域模式和特征的宝贵机会。主要挑战在于不同领域的时间序列数据特征存在显著差异,例如通道数量和时域分辨率尺度的变化。为应对这一挑战,我们提出了CrossTimeNet,一种新颖的跨域SSL学习框架,旨在从各个领域学习可迁移知识,从而大幅惠及目标下游任务。CrossTimeNet的关键特征之一是新设计的时间序列分词模块,该模块基于重构优化过程,能有效将原始时间序列转换为离散令牌序列。此外,我们强调,在SSL预训练期间预测高比例的损坏令牌对于提取跨域信息模式非常有益,而这一点在过去几年中很大程度上被忽视了。再者,与以往工作不同,我们的工作将预训练语言模型(PLM)作为编码器网络的初始化,探究将PLM学到的知识迁移到时间序列领域的可行性。通过这些努力,可以有效铺平通用时间序列模型跨域预训练的路径。我们在真实世界场景中,跨多个时间序列分类领域进行了大量实验。实验结果明确证实了CrossTimeNet的优越性能。

英文摘要:

Advancements in self-supervised pre-training (SSL) have significantly advanced the field of learning transferable time series representations, which can be very useful in enhancing the downstream task. Despite being effective, most existing works struggle to achieve cross-domain SSL pre-training, missing valuable opportunities to integrate patterns and features from different domains. The main challenge lies in the significant differences in the characteristics of time-series data across different domains, such as variations in the number of channels and temporal resolution scales. To address this challenge, we propose CrossTimeNet, a novel cross-domain SSL learning framework to learn transferable knowledge from various domains to largely benefit the target downstream task. One of the key characteristics of CrossTimeNet is the newly designed time series tokenization module, which could effectively convert the raw time series into a sequence of discrete tokens based on a reconstruction optimization process. Besides, we highlight that predicting a high proportion of corrupted tokens can be very helpful for extracting informative patterns across different domains during SSL pre-training, which has been largely overlooked in past years. Furthermore, unlike previous works, our work treats the pre-training language model (PLM) as the initialization of the encoder network, investigating the feasibility of transferring the knowledge learned by the PLM to the time series area. Through these efforts, the path to cross-domain pre-training of a generic time series model can be effectively paved. We conduct extensive experiments in a real-world scenario across various time series classification domains. The experimental results clearly confirm CrossTimeNet's superior performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑