Knowledge distillation is a popular technique for training a small student network to emulate a larger teacher model, such as an ensemble of networks. We show that while knowledge distillation can improve student generalization, it does not typically work as it is commonly understood: there often remains a surprisingly large discrepancy between the predictive distributions of the teacher and the student, even in cases when the student has the capacity to perfectly match the teacher. We identify difficulties in optimization as a key reason for why the student is unable to match the teacher. We also show how the details of the dataset used for distillation play a role in how closely the student matches the teacher -- and that more closely matching the teacher paradoxically does not always lead to better student generalization.
Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?
LeapLab, Tsinghua University（清华大学LeapLab）；Qiuzhen College, Tsinghua University（清华大学求真书院）；Beihang University（北京航空航天大学）；National University of Singapore（新加坡国立大学）；The Chinese University of Hong Kong（香港中文大学）；E Fund Management Co., Ltd.（易方达基金管理有限公司）
Dept. Computing Science & Alberta Machine Intelligence Institute (Amii), University of Alberta（计算科学系及阿尔伯塔人工智能研究所（Amii），阿尔伯塔大学）；Canada CIFAR AI Chair, Amii（加拿大CIFAR人工智能主席，Amii）
Do Deep Convolutional Nets Really Need to be Deep and Convolutional?
UC Irvine（加州大学欧文分校）；University of Edinburgh（爱丁堡大学）；Ecole Polytechnique de Montreal（蒙特利尔理工学院）；University of Alberta（阿尔伯塔大学）；University of Washington（华盛顿大学）；Microsoft Research（微软研究院）
When and Why Does Multi-Agent Debate Fail and Does It Really Underperform?
The Chinese University of Hong Kong（香港中文大学）；RIKEN Center for Advanced Intelligence Project（日本理化学研究院高级智能项目中心）；Hong Kong Baptist University（香港 Baptist大学）；The University of Tokyo（东京大学）
Does Deep Learning REALLY Outperform Non-deep Machine Learning for Clinical Prediction on Physiological Time Series?
Ricoh Software Research Center (Beijing) Co., Ltd.（理光软件研究所（北京）有限公司）；Japan Advanced Institute of Science and Technology (JAIST)（日本北陆先端科学技术大学院大学（JAIST））；Yale University School of Medicine（耶鲁大学医学院）；The Second Xiangya Hospital, Central South University（中南大学湘雅二医院）
Does LLM Alignment Really Need Diversity? An Empirical Study of Adapting RLVR Methods for Moral Reasoning
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Microsoft Research（微软研究院）；University of Michigan（密歇根大学）；Shanghai Jiao Tong University（上海交通大学）；CUHKSZ（香港中文大学（深圳））；THU（清华大学）
