This research investigates the enhancement of knowledge distillation (KD) processes in pre-trained models, an emerging field in knowledge transfer with significant implications for distributed training and federated learning environments. These environments benefit from reduced communication demands and accommodate various model architectures. Despite the adoption of numerous KD approaches for transferring knowledge among pre-trained models, a comprehensive understanding of KD's application in these scenarios is lacking. Our study conducts an extensive comparison of multiple KD techniques, including standard KD, tuned KD (via optimized temperature and weight parameters), deep mutual learning, and data partitioning KD. We assess these methods across various data distribution strategies to identify the most effective contexts for each. Through detailed examination of hyperparameter tuning, informed by extensive grid search evaluations, we pinpoint when adjustments are crucial to enhance model performance. This paper sheds light on optimal hyperparameter settings for distinct data partitioning scenarios and investigates KD's role in improving federated learning by minimizing communication rounds and expediting the training process. By filling a notable void in current research, our findings serve as a practical framework for leveraging KD in pre-trained models within collaborative and federated learning frameworks.
Rethinking Decoupled Knowledge Distillation: A Predictive Distribution Perspective
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University（数据科学与人工智能系，香港理工大学）；Department of Computing, The Hong Kong Polytechnic University（计算系，香港理工大学）；The Hong Kong Polytechnic University Shenzhen Research Institute（香港理工大学深圳研究院）
CAKD: A Correlation-Aware Knowledge Distillation Framework Based on Decoupling Kullback-Leibler Divergence
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Faculty of Engineering, The University of Sydney（悉尼大学工程学院）；School of Mechanical and Electronic Engineering, Wuhan University of Technology（武汉理工大学机电工程学院）
Jiangsu University（江苏大学）；Southwest University（西南大学）；The University of Sydney（悉尼大学）；Monash University（莫纳什大学）；Sichuan University（四川大学）；Nanyang Technological University（南洋理工大学）
Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models
Pingtan Research Institute of Xiamen University（厦门大学滨海研究院）；Imperial College London（伦敦帝国理工学院）；University of Sussex（苏塞克斯大学）；Purdue University（普渡大学）；University of Liverpool（利物浦大学）；JTB Technology Corp.（JTB技术公司）；Emory University（埃默里大学）；The University of Texas at Dallas（德克萨斯大学达拉斯分校）；Kyoto University（京都大学）；AppCubic；Georgia Institute of Technology（佐治亚理工学院）；AI Agent Lab（AI代理实验室）
Rethinking Reverse KL as Adaptive Entropy Distillation
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；School of Science and Technology, Hong Kong Metropolitan University（香港都会大学科技学院）
Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
University of Georgia（佐治亚大学）；The University of Texas at Arlington（德克萨斯大学阿灵顿分校）；Harvard University（哈佛大学）；Carnegie Mellon University（卡内基梅隆大学）；Vanderbilt University（范德堡大学）；Mayo Clinic Arizona（亚利桑那梅奥诊所）；Augusta University（奥古斯塔大学）
Decoupling Dark Knowledge via Block-wise Logit Distillation for Feature-level Alignment
College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息与电子工程学院）；ZJU-UIUC Institute, Zhejiang University（浙江大学-伊利诺伊大学厄巴纳-香槟分校联合学院）
DeepSalt: Bridging Laboratory and Satellite Spectra through Domain Adaptation and Knowledge Distillation for Large-Scale Soil Salinity Estimation
Department of Computer Science, Colorado State University, USA（计算机科学系，科罗拉多州立大学）；Department of Soil and Crop Sciences, Colorado State University, USA（土壤与作物科学系，科罗拉多州立大学）
Summarize-Exemplify-Reflect: Data-driven Insight Distillation Empowers LLMs for Few-shot Tabular Classification
ETH Zürich（苏黎世联邦理工学院）；University of Copenhagen（哥本哈根大学）；University of Science and Technology of China（中国科学技术大学）；University of Amsterdam（阿姆斯特丹大学）；East China Normal University（东华大学）
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Cambridge（剑桥大学）；Facebook AI Research（Facebook人工智能研究院）；Columbia University（哥伦比亚大学）
Stabilizing Consistency Training: A Flow Map Analysis and Self-Distillation
Department of Computer Science and Engineering（计算机科学与工程系）；Seoul National University（首尔国立大学）；Interdisciplinary Program in Artificial Intelligence（人工智能跨学科项目）
Scalable Model Merging with Progressive Layer-wise Distillation
Institute for Interdisciplinary Information Sciences, Tsinghua University（清华大学交叉信息研究院）；School of Computer Science, Beijing Institute of Technology（北京理工大学计算机学院）；Shanghai Qizhi Institute（上海启智研究院）
