超越对数似然:面向模型能力连续体的监督微调概率目标
Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuum
AI总结:
本文系统研究监督微调中不同概率目标函数,发现模型能力连续体决定目标优劣:强模型端偏好先验倾向目标(如-p),弱模型端对数似然最优,并理论解释其转换机制。
摘要:
监督微调(SFT)是后训练大型语言模型(LLM)的标准方法,但通常表现出有限的泛化能力。我们将此限制归因于其默认训练目标:负对数似然(NLL)。虽然NLL在从头训练时经典最优,但后训练处于不同范式,可能违反其最优性假设,因为模型已编码任务相关先验,且监督可能冗长且有噪声。在这项工作中,我们系统研究了各种基于概率的目标,并刻画了不同目标在不同条件下成功或失败的时间和原因。通过在8个模型骨干、27个基准和7个领域上的全面实验和广泛消融研究,我们揭示了控制目标行为的关键维度:模型能力连续体。在模型强端附近,降低低概率令牌权重的先验倾向目标(例如,-p, -p^{10}, 阈值变体)一致优于NLL;在模型弱端,NLL占主导;在中间,没有单一目标普遍最优。我们的理论分析进一步阐明了目标如何在连续体上交换位置,为根据模型能力调整目标提供了原则性基础。代码可在 https://github.com/GaotangLi/Beyond-Log-Likelihood 获取。
英文摘要:
Supervised fine-tuning (SFT) is the standard approach for post-training large language models (LLMs), yet it often shows limited generalization. We trace this limitation to its default training objective: negative log likelihood (NLL). While NLL is classically optimal when training from scratch, post-training operates in a different paradigm and could violate its optimality assumptions, where models already encode task-relevant priors and supervision can be long and noisy. In this work, we systematically study various probability-based objectives and characterize when and why different objectives succeed or fail under varying conditions. Through comprehensive experiments and extensive ablation studies across 8 model backbones, 27 benchmarks, and 7 domains, we uncover a critical dimension that governs objective behavior: the model-capability continuum. Near the model-strong end, prior-leaning objectives that downweight low-probability tokens (e.g., $-p$, $-p^{10}$, thresholded variants) consistently outperform NLL; toward the model-weak end, NLL dominates; in between, no single objective prevails. Our theoretical analysis further elucidates how objectives trade places across the continuum, providing a principled foundation for adapting objectives to model capability. The code is available at https://github.com/GaotangLi/Beyond-Log-Likelihood.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。