arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2307.03347cs.LG
收藏

面向时间序列数据的跨域模型压缩的通用与联合知识蒸馏

Distilling Universal and Joint Knowledge for Cross-Domain Model Compression on Time Series Data

Institute for Infocomm Research, A*STAR(新加坡科技研究局资讯通信研究院);Nanyang Technological University(南洋理工大学);Centre for Frontier AI Research, A*STAR(新加坡科技研究局前沿人工智能研究中心)

Qing Xu, Min Wu, Xiaoli Li, Kezhi Mao, Zhenghua Chen

AI总结:

本文提出UNI-KD端到端框架,通过对抗学习同时迁移跨域通用特征级知识和联合logit级知识,解决时间序列模型跨域压缩中的域偏移问题,并在四个数据集上验证其优于SOTA方法。

摘要:

在许多现实世界的时间序列任务中,主流深度学习模型的计算复杂度往往阻碍其在资源受限环境(如智能手机)中的部署。此外,由于模型训练(源域)与部署(目标域)阶段之间不可避免的域偏移,在跨域场景下压缩这些深度模型变得更加困难。尽管已有一些工作探索了用于模型压缩的跨域知识蒸馏,但它们要么偏向源数据,要么在源数据和目标数据之间严重纠缠。为此,我们设计了一种新颖的端到端框架,称为通用与联合知识蒸馏(UNI-KD),用于跨域模型压缩。具体而言,我们提出通过对抗学习方案,将跨源域和目标域的通用特征级知识以及两个域共享的联合logit级知识从教师模型迁移到学生模型。更具体地说,采用特征域判别器对齐教师和学生的表示,以实现通用知识迁移;利用数据域判别器优先选择域共享样本,以实现联合知识迁移。在四个时间序列数据集上的大量实验结果表明,我们提出的方法优于最先进(SOTA)基准。

英文摘要:

For many real-world time series tasks, the computational complexity of prevalent deep leaning models often hinders the deployment on resource-limited environments (e.g., smartphones). Moreover, due to the inevitable domain shift between model training (source) and deploying (target) stages, compressing those deep models under cross-domain scenarios becomes more challenging. Although some of existing works have already explored cross-domain knowledge distillation for model compression, they are either biased to source data or heavily tangled between source and target data. To this end, we design a novel end-to-end framework called Universal and joint knowledge distillation (UNI-KD) for cross-domain model compression. In particular, we propose to transfer both the universal feature-level knowledge across source and target domains and the joint logit-level knowledge shared by both domains from the teacher to the student model via an adversarial learning scheme. More specifically, a feature-domain discriminator is employed to align teacher's and student's representations for universal knowledge transfer. A data-domain discriminator is utilized to prioritize the domain-shared samples for joint knowledge transfer. Extensive experimental results on four time series datasets demonstrate the superiority of our proposed method over state-of-the-art (SOTA) benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑