Knowledge distillation is the technique of compressing a larger neural network, known as the teacher, into a smaller neural network, known as the student, while still trying to maintain the performance of the larger neural network as much as possible. Existing methods of knowledge distillation are mostly applicable for classification tasks. Many of them also require access to the data used to train the teacher model. To address the problem of knowledge distillation for regression tasks under the absence of original training data, previous work has proposed a data-free knowledge distillation method where synthetic data are generated using a generator model trained adversarially against the student model. These synthetic data and their labels predicted by the teacher model are then used to train the student model. In this study, we investigate the behavior of various synthetic data generation methods and propose a new synthetic data generation strategy that directly optimizes for a large but bounded difference between the student and teacher model. Our results on benchmark and case study experiments demonstrate that the proposed strategy allows the student model to learn better and emulate the performance of the teacher model more closely.
Toward Robust Semi-supervised Regression via Dual-stream Knowledge Distillation
Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences(重庆绿色智能技术研究所,中国科学院);Chongqing School, University of Chinese Academy of Sciences(中国科学院大学重庆学院);Singapore Management University(新加坡管理大学);Beijing University of Posts and Telecommunications(北京邮电大学)
Data-Efficient Symbolic Regression via Foundation Model Distillation
Institute for Clarity in Documentation(文档清晰研究所);Inria Paris-Rocquencourt(巴黎-罗克琴特研究所);Rajiv Gandhi University(拉吉夫·甘地大学);Tsinghua University(清华大学);Palmer Research Laboratories(帕尔默研究实验室);Arizona State University(亚利桑那州立大学);Clemson University(克莱姆森大学)
Cross-Paradigm Knowledge Distillation: A Comprehensive Study of Bidirectional Transfer Between Random Forests and Deep Neural Networks for Big Data Applications
SymbolicArena: A Unified Infrastructure for Benchmark Distillation and Dynamic Evaluation in Symbolic Regression
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);CDL, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所CDL);Nanjing University of Science and Technology(南京理工大学);School of Future Technology, University of Chinese Academy of Sciences(中国科学院大学未来技术学院)
Sequential Subset Matching for Dataset Distillation
Centre for Frontier AI Research (CFAR), Agency for Science, Technology and Research (A*STAR), Singapore(前沿人工智能研究中心(CFAR),新加坡科技研究局(A*STAR));Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR), Singapore(高性能计算研究所(IHPC),新加坡科技研究局(A*STAR));Department of Statistics, Purdue University(普渡大学统计系)
Breaking Class Barriers: Efficient Dataset Distillation via Inter-Class Feature Compensator
Centre for Frontier AI Research, Agency for Science, Technology and Research(新加坡科技研究局前沿人工智能研究中心);Institute of High Performance Computing, Agency for Science, Technology and Research(新加坡科技研究局高性能计算研究所);University of Nevada, Reno(内华达大学雷诺分校)
State Diversity Matters in Offline Behavior Distillation
School of Computer Science, Faculty of Engineering, The University of Sydney(悉尼大学计算机科学学院,工程学院);Generative AI Lab, College of Computing and Data Science, Nanyang Technological University(南洋理工大学生成人工智能实验室,计算与数据科学学院)
Accelerating Dataset Distillation via Model Augmentation
Zhejiang University(浙江大学);Texas A&M University(德州农工大学);Microsoft Research(微软研究院);New York University(纽约大学);Beijing Academy of Artificial Intelligence(北京智源人工智能研究院);University of Connecticut(康涅狄格大学);University of North Carolina, Chapel Hill(北卡罗来纳大学教堂山分校);North Carolina State University(北卡罗来纳州立大学)
Recover-LoRA for Aggressive Quantization: Reclaiming Accuracy in 2-Bit Language Models via Low-Rank Adaptation with Knowledge Distillation on Synthetic Data
Privacy-Preserving Model Transcription with Differentially Private Synthetic Distillation
Institute of Information Engineering at Chinese Academy of Sciences(中国科学院信息工程研究所);Beijing Institute of Astronautical Systems Engineering(北京航天系统工程研究所);Trustworthy Machine Learning Lab, School of Computer Science, The University of Sydney(悉尼大学计算机科学学院可信机器学习实验室)
Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustment
Centre for Frontier AI Research (CFAR)(前沿人工智能研究中心);Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局);Institute of High Performance Computing(高性能计算研究所);XiDian University(西安电子科技大学);National University of Singapore(新加坡国立大学);Hubei University(湖北大学)
Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation
Centre for Frontier AI Research (CFAR), Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局前沿人工智能研究中心(CFAR));National University of Singapore(新加坡国立大学);Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局高性能计算研究所(IHPC));The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))