In learning-to-rank problems, a privileged feature is one that is available during model training, but not available at test time. Such features naturally arise in merchandised recommendation systems; for instance, "user clicked this item" as a feature is predictive of "user purchased this item" in the offline data, but is clearly not available during online serving. Another source of privileged features is those that are too expensive to compute online but feasible to be added offline. Privileged features distillation (PFD) refers to a natural idea: train a "teacher" model using all features (including privileged ones) and then use it to train a "student" model that does not use the privileged features.
In this paper, we first study PFD empirically on three public ranking datasets and an industrial-scale ranking problem derived from Amazon's logs. We show that PFD outperforms several baselines (no-distillation, pretraining-finetuning, self-distillation, and generalized distillation) on all these datasets. Next, we analyze why and when PFD performs well via both empirical ablation studies and theoretical analysis for linear models. Both investigations uncover an interesting non-monotone behavior: as the predictive power of a privileged feature increases, the performance of the resulting student model initially increases but then decreases. We show the reason for the later decreasing performance is that a very predictive privileged teacher produces predictions with high variance, which lead to high variance student estimates and inferior testing performance.
Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);University of Cambridge(剑桥大学);Facebook AI Research(Facebook人工智能研究院);Columbia University(哥伦比亚大学)
PR-OPD: Privileged Representation On-policy Self-Distillation for Agentic Reinforcement Learning
University of Florida(佛罗里达大学);University of Illinois at Chicago(伊利诺伊大学芝加哥分校);Case Western Reserve University(凯斯西储大学);Vanderbilt University(范德堡大学);Weill Cornell Medicine(威尔康奈尔医学院)
$π$-Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Data
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉学科学院);Meituan(美团);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Longitudinal Risk Prediction in Mammography with Privileged History Distillation
LIVIA, ILLS, Systems Engineering Dept., ETS Montreal, Canada(LIVIA、ILLs系统工程系、蒙特利尔大学ETS、加拿大);Goodman Cancer Institute, Dept. of Oncology, McGill University, Canada(Goodman癌症研究所、肿瘤学系、麦吉尔大学、加拿大)
BLEND: Behavior-guided Neural Population Dynamics Modeling via Privileged Knowledge Distillation
The Hong Kong University of Science and Technology(香港科技大学);Beijing Institute of Collaborative Innovation(北京协同创新研究院);Peking University(北京大学);Beihang University(北京航空航天大学)
SCOPE-OPSD: Fisher-Conditioned Privileged Subspaces for On-Policy Self-Distillation
Chongqing Ant Consumer Finance Co., Ltd.(重庆蚂蚁消费金融有限公司);Alibaba Cloud Computing Co., Ltd.(阿里云计算有限公司);Chongqing University of Posts and Telecommunications(重庆邮电大学)
Learning Individual Scientific Taste with Privileged Hindsight Distillation
Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Yau Mathematical Sciences Center, Tsinghua University(清华大学丘成桐数学科学中心);Qiuzhen College, Tsinghua University(清华大学邱士栋书院)
TriDeNT: Triple Deep Network Training for Privileged Knowledge Distillation in Histopathology
School of Cancer Sciences, University of Glasgow(格拉斯哥大学癌症科学学院);Cancer Research UK Scotland Institute(英国癌症研究苏格兰研究所);School of Computing Science, University of Glasgow(格拉斯哥大学计算科学学院);School of Mathematics and Statistics, University of Glasgow(格拉斯哥大学数学与统计学院);Division of Biosciences, University College London(伦敦大学学院生物科学部)
City University of Hong Kong(香港城市大学);Tsinghua University(清华大学);Shenzhen University of Advanced Technology(深圳理工大学);Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))
Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences(中国科学院大学杭州高等研究院);The Chinese University of Hong Kong(香港中文大学);University of Science and Technology of China(中国科学技术大学);University of Chinese Academy of Sciences(中国科学院大学);vivo AI Lab(vivo AI实验室)