发表机构
University of California, Davis（加州大学戴维斯分校）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究通过分类任务案例，理论分析了SFT与RFT在语义遗忘上的差异，发现SFT因风格漂移导致遗忘，而RFT保持语义不变，并用模拟验证了该分离现象。
AI 中文摘要
为什么监督微调（SFT）即使所有教师示范在语义上都是正确的，也会比强化微调（RFT）导致更多的遗忘？我们在分类任务上研究这个问题，其中每个语义类内的词元以不同风格表达相同的语义答案。这些任务共享一个潜在的语义规则，但在提示分布和教师的风格偏好上有所不同。利用一个易于处理的线性-softmax策略，我们将更新精确分解为语义和风格两个组成部分。我们表明，在共同的策略和提示下，SFT和RFT具有平行的语义更新，但它们的风格动态不同。从没有类内风格偏好的策略开始，使用精确策略梯度的RFT保持这种对称性，而使用非均匀教师的SFT在总体更新下沿非零任务均值发展出离轴风格漂移。我们利用这种漂移在明确条件下建立分离：对于从共同的完美拟合检查点进行的总体更新，SFT遗忘在有限训练区间内具有严格正的下界，而RFT保持零语义错误。任务序列上的模拟支持这些理论预测。
英文摘要
Why does supervised fine-tuning (SFT) lead to more forgetting than reinforcement fine-tuning (RFT), even when all teacher demonstrations are semantically correct? We study this question on classification tasks where tokens within each semantic class express the same semantic answer in different styles. The tasks share an underlying semantic rule but differ in their prompt distributions and teachers' stylistic preferences. Using a tractable linear-softmax policy, we derive an exact decomposition of the updates into semantic and style components. We show that, at a common policy and prompt, SFT and RFT have parallel semantic updates but differ in their style dynamics. Starting from a policy with no within-class style preference, RFT with exact policy gradients preserves this symmetry, whereas SFT with a nonuniform teacher develops off-axis style drift along a nonzero task mean under population updates. We use this drift to establish a separation under explicit conditions: for population updates from a common perfectly fitted checkpoint, SFT forgetting admits a strictly positive lower bound over a finite training interval, while RFT retains zero semantic error. Simulations over task sequences support these theoretical predictions.
Comments43 pages, 7 figures
