熵适应微调：解决自信冲突以缓解遗忘
Entropy-Adaptive Fine-Tuning: Resolving Confident Conflicts to Mitigate Forgetting
AI总结：
熵适应微调通过利用令牌级熵区分知识不确定性和冲突，缓解微调过程中的遗忘问题，提升模型在多个领域的表现。
摘要：
监督微调（SFT）是领域适应的标准范式，但经常导致灾难性遗忘的成本。与之形成鲜明对比的是，在线强化学习（RL）能有效保持一般能力。我们研究这一差异并发现了一个根本的分布性差距：虽然RL与模型的内部信念一致，SFT迫使模型适应外部监督。这种不匹配常常表现为“自信冲突”标记，这些标记具有低概率但低熵的特征。在这些情况下，模型对其预测非常自信，但被迫学习一个分歧的真实值，从而触发破坏性的梯度更新。为了解决这一问题，我们提出了熵适应微调（EAFT）。与仅依赖预测概率的方法不同，EAFT利用令牌级熵作为门控机制，以区分知识冲突和知识不确定性。这使模型能够在不确定的样本上学习，同时抑制冲突数据上的梯度。在Qwen和GLM系列（从4B到32B参数）上广泛实验，涵盖数学、医学和代理领域，证实了我们的假设。EAFT在保持下游性能与标准SFT一致的同时，显著减轻了一般能力的退化。
英文摘要：
Supervised Fine-Tuning (SFT) is the standard paradigm for domain adaptation, yet it frequently incurs the cost of catastrophic forgetting. In sharp contrast, on-policy Reinforcement Learning (RL) effectively preserves general capabilities. We investigate this discrepancy and identify a fundamental distributional gap: while RL aligns with the model's internal belief, SFT forces the model to fit external supervision. This mismatch often manifests as "Confident Conflicts" tokens characterized by low probability but low entropy. In these instances, the model is highly confident in its own prediction but is forced to learn a divergent ground truth, triggering destructive gradient updates. To address this, we propose Entropy-Adaptive Fine-Tuning (EAFT). Unlike methods relying solely on prediction probability, EAFT utilizes token-level entropy as a gating mechanism to distinguish between epistemic uncertainty and knowledge conflict. This allows the model to learn from uncertain samples while suppressing gradients on conflicting data. Extensive experiments on Qwen and GLM series (ranging from 4B to 32B parameters) across mathematical, medical, and agentic domains confirm our hypothesis. EAFT consistently matches the downstream performance of standard SFT while significantly mitigating the degradation of general capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
