SED-SFT: 在监督微调中选择性鼓励多样性
SED-SFT: Selectively Encouraging Diversity in Supervised Fine-Tuning
AI总结：
SED-SFT通过引入选择性熵正则化和掩码机制，在监督微调中提升生成多样性，从而提高后续强化学习的性能。
摘要：
监督微调（SFT）后接强化学习（RL）已成为大语言模型（LLMs）训练后的标准范式。然而，传统的SFT过程由交叉熵（CE）损失驱动，往往导致模式崩溃，即模型过度集中在特定的响应模式上。这种分布多样性不足严重限制了后续RL所需的探索效率。尽管近期研究尝试通过替换CE损失来改进SFT，旨在保留多样性或优化更新策略，但它们未能充分平衡多样性和准确性，导致RL后的性能不理想。为了解决模式崩溃问题，我们提出了SED-SFT，该方法基于令牌探索空间适应性地鼓励多样性。该框架将选择性熵正则化项和选择性掩码机制引入优化目标中。在八个数学基准测试中进行的广泛实验表明，SED-SFT在几乎不增加计算开销的情况下显著提高了生成多样性，相对于CE损失，在Llama-3.2-3B-Instruct和Qwen2.5-Math-7B-Instruct上的后续RL性能平均提高了2.06和1.20个点。代码可在https://github.com/pppa2019/SED-SFT上公开获取。
英文摘要：
Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL) has emerged as the standard post-training paradigm for large language models (LLMs). However, the conventional SFT process, driven by Cross-Entropy (CE) loss, often induces mode collapse, where models over-concentrate on specific response patterns. This lack of distributional diversity severely restricts the exploration efficiency required for subsequent RL. While recent studies have attempted to improve SFT by replacing the CE loss, aiming to preserve diversity or refine the update policy, they fail to adequately balance diversity and accuracy, thereby yielding suboptimal performance after RL. To address the mode collapse problem, we propose SED-SFT, which adaptively encourages diversity based on the token exploration space. This framework introduces a selective entropy regularization term with a selective masking mechanism into the optimization objective. Extensive experiments across eight mathematical benchmarks demonstrate that SED-SFT significantly enhances generation diversity with a negligible computational overhead increase compared with CE loss, yielding average improvements of 2.06 and 1.20 points in subsequent RL performance over standard CE-based baselines on Llama-3.2-3B-Instruct and Qwen2.5-Math-7B-Instruct, respectively. The code is publicly available at https://github.com/pppa2019/SED-SFT
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
