arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2205.09928cs.LG
收藏

基于交叉重建Transformer的自监督时间序列表示学习

Self-Supervised Time Series Representation Learning via Cross Reconstruction Transformer

National Institute of Health Data Science, Peking University(北京大学健康医疗大数据国家研究院);Institute of Medical Technology, Health Science Center of Peking University(北京大学医学部医学技术研究院);National University of Singapore(新加坡国立大学);HeartVoice Medical Technology(心之声医疗科技)

Wenrui Zhang, Ling Yang, Shijia Geng, Shenda Hong

AI总结:

本文提出交叉重建Transformer(CRT),通过时频域随机丢弃与跨域重建实现自监督时间序列表示学习,并引入实例判别约束和课程学习策略以提升表示判别性与训练稳定性。

摘要:

在现实场景中,带标签样本通常较为稀缺,因此时间序列中的无监督/自监督表示学习至关重要。现有方法主要利用对比学习框架,该框架自动学习理解相似和不相似的数据对。然而,这些方法受限于构建数据对的先验知识、繁琐的采样策略,以及在遇到采样偏差时性能不稳定。此外,很少有工作关注有效建模时间-频谱关系以扩展表示能力。本文旨在从一个新的视角学习时间序列表示,并提出交叉重建Transformer(CRT),以统一的方式解决上述问题。CRT通过跨域丢弃-重建任务实现时间序列表示学习。具体而言,我们将时间序列变换到频域,并在时域和频域中随机丢弃某些部分。与裁剪和掩码相比,丢弃能够最大程度地保留全局上下文。然后利用Transformer架构,通过在两个域中重建数据来充分捕获时间与频谱信息之间的跨域相关性,这被称为丢弃时间-频谱建模。为了在全局潜在空间中区分表示,我们提出实例判别约束,以减少不同时间序列之间的互信息并锐化决策边界。此外,我们提出了一种特定的课程学习策略来优化CRT,在训练过程中逐步增加丢弃比例。

英文摘要:

Unsupervised/self-supervised representation learning in time series is critical since labeled samples are usually scarce in real-world scenarios. Existing approaches mainly leverage the contrastive learning framework, which automatically learns to understand the similar and dissimilar data pairs. Nevertheless, they are restricted to the prior knowledge of constructing pairs, cumbersome sampling policy, and unstable performances when encountering sampling bias. Also, few works have focused on effectively modeling across temporal-spectral relations to extend the capacity of representations. In this paper, we aim at learning representations for time series from a new perspective and propose Cross Reconstruction Transformer (CRT) to solve the aforementioned problems in a unified way. CRT achieves time series representation learning through a cross-domain dropping-reconstruction task. Specifically, we transform time series into the frequency domain and randomly drop certain parts in both time and frequency domains. Dropping can maximally preserve the global context compared to cropping and masking. Then a transformer architecture is utilized to adequately capture the cross-domain correlations between temporal and spectral information through reconstructing data in both domains, which is called Dropped Temporal-Spectral Modeling. To discriminate the representations in global latent space, we propose Instance Discrimination Constraint to reduce the mutual information between different time series and sharpen the decision boundaries. Additionally, we propose a specified curriculum learning strategy to optimize the CRT, which progressively increases the dropping ratio in the training process.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑