Advancements in self-supervised pre-training (SSL) have significantly advanced the field of learning transferable time series representations, which can be very useful in enhancing the downstream task. Despite being effective, most existing works struggle to achieve cross-domain SSL pre-training, missing valuable opportunities to integrate patterns and features from different domains. The main challenge lies in the significant differences in the characteristics of time-series data across different domains, such as variations in the number of channels and temporal resolution scales. To address this challenge, we propose CrossTimeNet, a novel cross-domain SSL learning framework to learn transferable knowledge from various domains to largely benefit the target downstream task. One of the key characteristics of CrossTimeNet is the newly designed time series tokenization module, which could effectively convert the raw time series into a sequence of discrete tokens based on a reconstruction optimization process. Besides, we highlight that predicting a high proportion of corrupted tokens can be very helpful for extracting informative patterns across different domains during SSL pre-training, which has been largely overlooked in past years. Furthermore, unlike previous works, our work treats the pre-training language model (PLM) as the initialization of the encoder network, investigating the feasibility of transferring the knowledge learned by the PLM to the time series area. Through these efforts, the path to cross-domain pre-training of a generic time series model can be effectively paved. We conduct extensive experiments in a real-world scenario across various time series classification domains. The experimental results clearly confirm CrossTimeNet's superior performance.
Intra-domain and cross-domain transfer learning for time series data -- How transferable are the features?
University of Rijeka(里耶卡大学);Istituto Nazionale di Geofisica e Vulcanologia(国家地球物理与火山研究所);Roma Tre University(罗马第三大学);Center for Artificial Intelligence and Cybersecurity, University of Rijeka(里耶卡大学人工智能与网络安全中心)
Learning Transferable Policies from Action-free Time Series Through Dynamical Embeddings
Hector Institute for AI in Psychiatry (HITKIP)(赫克托人工智能精神病学研究所);Central Institute of Mental Health (CIMH)(中央精神健康研究所);Heidelberg University(海德堡大学);Interdisciplinary Center for Scientific Computing (IWR)(跨学科科学计算中心);Hertie Institute for AI in Brain Health(赫蒂人工智能脑健康研究所);University of Tübingen(蒂宾根大学)
Noise May Contain Transferable Knowledge: Understanding Semi-supervised Heterogeneous Domain Adaptation from an Empirical Perspective
Beijing Teleinfo Technology Company Ltd.(北京 Teleinfo 科技有限公司);China Academy of Information and Communications Technology(中国信息通信研究院);Inspur Computer Technology Co., Ltd.(浪潮计算机技术有限公司);Southern University of Science and Technology(南方科技大学);Research Institute of Industrial Internet of Things(工业互联网物联网研究院);Hong Kong University of Science and Technology(香港科技大学);WeBank(微众银行)
Pre-training Epidemic Time Series Forecasters with Compartmental Prototypes
Emory University Department of Computer Science Atlanta GA USA(埃默里大学计算机科学系);Emory University Rollins School of Public Health Atlanta GA USA(埃默里大学罗林斯公共卫生学院);Emory University(埃默里大学)
Olivia: Harmonizing Time Series Foundation Models with Power Spectral Density
Beijing Institute of Technology(北京理工大学);North China Institute of Computing Technology(华北计算技术研究所);State Information Center(国家信息中心);University of Auckland(奥克兰大学);Northwest Polytechnical University(西北工业大学);Victoria University of Wellington(威灵顿维多利亚大学)
Lightweight Time Series Data Valuation on Time Series Foundation Models via In-Context Finetuning
Sun Yat-sen University State Key Laboratory of Internet Architecture(中山大学互联网架构国家重点实验室);Tsinghua University(清华大学);National University of Singapore(新加坡国立大学)
STEP: Scientific Time-Series Encoder Pretraining via Cross-Domain Distillation
Shanghai Aritificial Intelligence Laboratory(上海人工智能实验室);Tsinghua University(清华大学);Harbin Institute of Technology(哈尔滨工业大学);Sichuan University(四川大学);University of Cambridge(剑桥大学)
ChorusTIC: Training-Free Multivariate Time Series Classification via Chorus In-Context Learning
Guangdong University of Technology(广东工业大学);Université Paris Cité(巴黎西岱大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Shantou University(汕头大学)
Generalized and Transferable Patient Language Representation for Phenotyping with Limited Data
The University of Texas Health Science Center at Houston(德克萨斯大学休斯顿健康科学中心);McGovern Medical School, The University of Texas Health Science Center at Houston(德克萨斯大学休斯顿健康科学中心麦戈文医学院)
Transferable Delay-Aware Reinforcement Learning via Implicit Causal Graph Modeling
College of Computer Science and Technology, National University of Defense Technology(计算机科学与技术学院,国防科技大学);Intelligent Game and Decision Lab (IGDL)(智能游戏与决策实验室);Institute of Military Transportation(军事交通运输研究院);School of Artificial Intelligence, Hebei University of Technology(人工智能学院,河北工业大学)
Foundation Model for Cardiac Time Series via Masked Latent Attention
Department of Computer Science, ETH Zurich, Switzerland(苏黎世联邦理工学院计算机科学系,瑞士);Department of Biomedical Engineering, University of Basel, Switzerland(巴塞尔大学生物医学工程系,瑞士)