Preserving Diversity in Supervised Fine-Tuning of Large Language Models
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Research Institute of Big Data(深圳大数据研究院);Nanjing University(南京大学);Hong Kong University of Science and Technology(香港科技大学);University of Pennsylvania(宾夕法尼亚大学)
Large Language Models (LLMs) typically rely on Supervised Fine-Tuning (SFT) to specialize in downstream tasks, with the Cross Entropy (CE) loss being the de facto choice. However, CE maximizes the likelihood of observed data without accounting for alternative possibilities. As such, CE usually leads to reduced diversity in the model's outputs, which hinders further development that requires sampling to explore better responses. To address this limitation, this paper introduces a new game-theoretic formulation for SFT. In this framework, an auxiliary variable is introduced to regulate the learning process. We prove that the proposed game-theoretic approach connects to the problem of reverse KL minimization with entropy regularization. This regularization prevents over-memorization of training data and promotes output diversity. To implement this framework, we develop GEM, a new training algorithm that is computationally efficient as CE by leveraging some unique properties of LLMs. Empirical studies of pre-trained models from 3B to 70B parameters show that GEM achieves comparable downstream performance to CE while significantly enhancing output diversity. This increased diversity translates to performance gains in test-time compute scaling for chat and code generation tasks. Moreover, we observe that preserving output diversity has the added benefit of mitigating forgetting, as maintaining diverse outputs encourages models to retain pre-trained knowledge throughout the training process.
Mila - Quebec AI Institute(魁北克AI研究所);Polytechnique Montréal(蒙特利尔理工学院);Université de Montréal(蒙特利尔大学);McGill University(麦吉尔大学);CIFAR AI Chair(CIFAR人工智能主席);Google DeepMind(谷歌DeepMind)
From Random to Informed Data Selection: A Diversity-Based Approach to Optimize Human Annotation and Few-Shot Learning
Escola Politécnica, Universidade de São Paulo(圣保罗大学理工学院);Télécom Paris, Institut Polytechnique de Paris(巴黎高等电信学校,巴黎综合理工学院);Faculty of Economics, University of Porto(波尔图大学经济学院);INESC TEC(葡萄牙系统工程与计算机技术研究所)
Improving Semi-supervised Federated Learning by Reducing the Gradient Diversity of Models
Southeast University(东南大学);University of California, Berkeley(加州大学伯克利分校);University of Michigan, Ann Arbor(密歇根大学安娜堡分校);International Computer Science Institute(国际计算机科学研究所)
CentraleSupélec(中央高等电力学院);Université Paris-Saclay(巴黎-萨克雷大学);Institut de recherche technologique Saint Exupéry(圣埃克苏佩里技术研究院);Institut de mathématiques de Toulouse(图卢兹数学研究所);Université Paul Sabatier(保罗·萨巴蒂埃大学);CNRS(法国国家科学研究中心);HEC Paris(巴黎高等商学院)
Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNets
Mila, Université de Montréal(蒙特利尔大学Mila实验室);Max Planck Institute for Intelligent Systems - Tübingen(智能系统马克斯·普朗克研究所(图宾根));The Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳));University of Tübingen(图宾根大学);University of Cambridge(剑桥大学);Microsoft Research(微软研究院)
Beyond Instance Consistency: Investigating View Diversity in Self-supervised Learning
Institute for Infocomm Research (I 2 R), A*STAR, Singapore(新加坡*STAR信息与通信研究所);National University of Singapore, Singapore(新加坡国立大学);Sichuan University, China(四川大学);Southeast University, China(东南大学);Shanghai Jiaotong University, China(上海交通大学)
GraphSculptor: Sculpting Pre-training Coreset for Graph Self-supervised Learning
Sangfor Technologies Inc.(Sangfor 技术公司);Wuhan University(武汉大学);The University of Melbourne(墨尔本大学);Dalian Maritime University(大连海事大学);Hunan University of Technology and Business(湖南工业大学;湘江实验室);Xiangjiang Laboratory
Hydra: Preserving Ensemble Diversity for Model Distillation
Imperial College London(帝国理工学院);University of Amsterdam(阿姆斯特丹大学);ETH Zurich(苏黎世联邦理工学院);University of Warsaw(华沙大学);Google Research(谷歌研究院);University of California, Irvine(加州大学欧文分校)
DuSK: A Dual Structure-preserving Kernel for Supervised Tensor Learning with Applications to Neuroimages
South China University of Technology(华南理工大学);University of Illinois at Chicago(伊利诺伊大学芝加哥分校);Northwestern University(西北大学);Guangdong University of Technology(广东工业大学)
Diversified Ensemble of Independent Sub-Networks for Robust Self-Supervised Representation Learning
LMU Munich(慕尼黑大学);Munich Center for Machine Learning(慕尼黑机器学习中心);Institute of Statistics, LMU Munich(慕尼黑大学统计研究所);Institute of Informatics, LMU Munich(慕尼黑大学信息学研究所)
DyJR: Preserving Diversity in Reinforcement Learning with Verifiable Rewards via Dynamic Jensen-Shannon Replay
Griffith University, Australia(澳大利亚格里菲斯大学);Fudan University, China(复旦大学);Beijing University of Post(北京邮电大学);Shanghai Innovation Institute, China(上海创新研究院);Shanghai Academy of Artificial Intelligence for Science, China(上海人工智能科学研究院)
The Efficacy of Semantics-Preserving Transformations in Self-Supervised Learning for Medical Ultrasound
Cheriton School of Computer Science, University of Waterloo(滑铁卢大学计算机科学学院);Department of Systems Design Engineering, University of Waterloo(滑铁卢大学系统设计工程系);Schulich School of Medicine and Dentistry, Western University(西部大学施乐希医学与牙科学校)