Distilling Universal and Joint Knowledge for Cross-Domain Model Compression on Time Series Data
Institute for Infocomm Research, A*STAR(新加坡科技研究局资讯通信研究院);Nanyang Technological University(南洋理工大学);Centre for Frontier AI Research, A*STAR(新加坡科技研究局前沿人工智能研究中心)
Qing Xu, Min Wu, Xiaoli Li, Kezhi Mao, Zhenghua Chen
For many real-world time series tasks, the computational complexity of prevalent deep leaning models often hinders the deployment on resource-limited environments (e.g., smartphones). Moreover, due to the inevitable domain shift between model training (source) and deploying (target) stages, compressing those deep models under cross-domain scenarios becomes more challenging. Although some of existing works have already explored cross-domain knowledge distillation for model compression, they are either biased to source data or heavily tangled between source and target data. To this end, we design a novel end-to-end framework called Universal and joint knowledge distillation (UNI-KD) for cross-domain model compression. In particular, we propose to transfer both the universal feature-level knowledge across source and target domains and the joint logit-level knowledge shared by both domains from the teacher to the student model via an adversarial learning scheme. More specifically, a feature-domain discriminator is employed to align teacher's and student's representations for universal knowledge transfer. A data-domain discriminator is utilized to prioritize the domain-shared samples for joint knowledge transfer. Extensive experimental results on four time series datasets demonstrate the superiority of our proposed method over state-of-the-art (SOTA) benchmarks.
DLink: Distilling Layer-wise and Dominant Knowledge from EEG Foundation Models
Xiamen University(厦门大学);Nanyang Technological University(南洋理工大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Southeast University(东南大学)
Temporal Saliency-Guided Distillation: A Scalable Framework for Distilling Video Datasets
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Peng Cheng Laboratory(鹏城实验室);Tsinghua Shenzhen International Graduate School(清华大学深圳国际研究生院);Shanghai Jiao Tong University(上海交通大学)
Distilling Time Series Foundation Models for Efficient Forecasting
The City College of New York, City University of New York, USA(纽约城市学院,纽约城市大学);Institute of Computing Technology, Chinese Academy of Sciences, China(中国科学院计算技术研究所);Stevens Institute of Technology, USA(史蒂文斯理工学院)
A Joint Approach to Local Updating and Gradient Compression for Efficient Asynchronous Federated Learning
Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Tsinghua-Berkeley Shenzhen Institute, Tsinghua University(清华大学伯克利深圳研究院);Southern University of Science and Technology(南方科技大学);School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳)计算机科学与技术学院)
Hardware-Aware Deployment of Joint SAR Compression and Despeckling on FPGA
Technical University of Munich(慕尼黑工业大学);Remote Sensing Technology Institute, German Aerospace Center (DLR)(德国航空航天中心遥感技术研究所);University of the Bundeswehr Munich(慕尼黑联邦国防军大学)
Uncertainty quantification in model discovery by distilling interpretable material constitutive models from Gaussian process posteriors
Technische Universität Braunschweig, Institute of Applied Mechanics(Braunschweig 技术大学应用力学研究所);Technische Universität Braunschweig, Institute for Acoustics(Braunschweig 技术大学声学研究所)
Lossless Model Compression via Joint Low-Rank Factorization Optimization
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Peng Cheng Laboratory(鹏城实验室);the School of Microelectronics, South China University of Technology(华南理工大学微电子学院)