发表机构
School of Mathematical Sciences, Peking University; Institute of Computing Technology, Chinese Academy of Sciences; Institute of Automation, Chinese Academy of Sciences; College of Computing, Georgia Institute of Technology（北京大学数学科学学院; 中国科学院计算技术研究所; 中国科学院自动化研究所; 佐治亚理工学院计算学院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究监督微调中存在的问题，利用香农和雷尼熵分析揭示预训练模型的多模态熵结构，提出LP-SFT目标，在多领域实验中提升性能，平衡准确率和k准确率。
AI 中文摘要
监督微调是使预训练语言模型适应下游领域的标准方法，但常以牺牲已有能力为代价提升目标域行为。标准交叉熵微调只促进观察到的标签令牌，而我们利用熵分析提出LP-SFT，在多领域实验中取得更好平衡。
英文摘要
Supervised fine-tuning (SFT) is widely used to adapt pretrained language models to downstream domains, but can over-specialize the model and degrade its pre-existing capabilities. Standard cross-entropy drives probability mass onto the observed target token and can make plausible alternatives that the pretrained model itself endorsed vanish. Using Shannon and Renyi entropies, we show that pretrained next-token distributions exhibit a regular multimodal structure, with entropy peaks corresponding to different numbers of plausible alternatives. Motivated by this observation, we propose LP-SFT, a Local-Preserving Supervised Fine-Tuning objective with two key design principles: it removes the supervised token from a local top-$K$ candidate set to avoid conflict with cross-entropy, and applies a locally normalized KL loss to preserve relative preferences among the remaining non-label alternatives. Across mixed-domain and domain-specific fine-tuning experiments, LP-SFT consistently outperforms standard SFT and recent baselines in aggregate performance while keeping base-endorsed alternatives from vanishing, achieving a favorable balance between single-sample accuracy and finite-budget solution accessibility, as measured by pass@1 and pass@$k$, respectively, with only a modest increase in training cost.
Comments23 pages, 6 figures
