arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.16673cs.LGcs.AI
收藏

在大语言模型的监督微调中保持多样性

Preserving Diversity in Supervised Fine-Tuning of Large Language Models

The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Research Institute of Big Data(深圳大数据研究院);Nanjing University(南京大学);Hong Kong University of Science and Technology(香港科技大学);University of Pennsylvania(宾夕法尼亚大学)

Ziniu Li, Congliang Chen, Tian Xu, Zeyu Qin, Jiancong Xiao, Zhi-Quan Luo, Ruoyu Sun

AI总结:

本文针对监督微调中交叉熵损失导致LLM输出多样性降低的问题,提出一种基于博弈论且计算高效的训练算法GEM,在保持下游性能的同时显著提升输出多样性并缓解模型遗忘。

摘要:

大型语言模型(LLMs)通常依赖监督微调(SFT)来专精于下游任务,其中交叉熵(CE)损失是事实上的选择。然而,CE最大化了观测数据的似然而未考虑其他替代可能性。因此,CE通常会导致模型输出的多样性降低,这阻碍了需要通过采样来探索更好响应的进一步发展。为了解决这一局限性,本文为SFT引入了一种新的博弈论公式。在该框架中,引入了一个辅助变量来调节学习过程。我们证明,所提出的博弈论方法与带有熵正则化的反向KL最小化问题相关联。这种正则化防止了对训练数据的过度记忆并促进了输出多样性。为了实现这一框架,我们开发了GEM,这是一种新的训练算法,通过利用LLMs的一些独特特性,它在计算上与CE一样高效。对参数规模从3B到70B的预训练模型进行的实证研究表明,GEM实现了与CE相当的下游性能,同时显著增强了输出多样性。这种增加的多样性转化为在聊天和代码生成任务的测试时计算缩放中的性能提升。此外,我们观察到保持输出多样性具有缓解遗忘的额外好处,因为保持多样的输出鼓励模型在整个训练过程中保留预训练知识。

英文摘要:

Large Language Models (LLMs) typically rely on Supervised Fine-Tuning (SFT) to specialize in downstream tasks, with the Cross Entropy (CE) loss being the de facto choice. However, CE maximizes the likelihood of observed data without accounting for alternative possibilities. As such, CE usually leads to reduced diversity in the model's outputs, which hinders further development that requires sampling to explore better responses. To address this limitation, this paper introduces a new game-theoretic formulation for SFT. In this framework, an auxiliary variable is introduced to regulate the learning process. We prove that the proposed game-theoretic approach connects to the problem of reverse KL minimization with entropy regularization. This regularization prevents over-memorization of training data and promotes output diversity. To implement this framework, we develop GEM, a new training algorithm that is computationally efficient as CE by leveraging some unique properties of LLMs. Empirical studies of pre-trained models from 3B to 70B parameters show that GEM achieves comparable downstream performance to CE while significantly enhancing output diversity. This increased diversity translates to performance gains in test-time compute scaling for chat and code generation tasks. Moreover, we observe that preserving output diversity has the added benefit of mitigating forgetting, as maintaining diverse outputs encourages models to retain pre-trained knowledge throughout the training process.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑