灾难性遗忘的机制起源：为什么RL比SFT更好地保留电路？
Mechanistic origins of catastrophic forgetting: why RL preserves circuits better than SFT?
AI总结：
通过引入差异电路脆弱性指标，研究比较了强化学习与监督微调在大型语言模型微调中对内部计算电路的保留程度，发现RL虽任务适应较慢但能更好保留电路，从而减轻灾难性遗忘。
摘要：
微调大型语言模型（LLMs）经常导致先前能力的灾难性遗忘。最近的研究表明，强化学习（RL）比监督微调（SFT）更有效地保留先前能力，这归因于策略梯度更新更接近基础策略\cite{shenfeld2025rl}。我们将这种行为解释扩展到机制层面，并探究RL的优势是否通过内部计算电路的更强保留来体现。我们引入了差异电路脆弱性，一种头部级别的度量，用于衡量电路在微调下的退化程度，并将其用于比较RL和SFT在Qwen2.5-3B-Instruct适应科学问答任务上的表现。我们发现了清晰的机制权衡：SFT更快地适应目标任务，但导致更大的电路破坏和先前能力的遗忘，而RL保留了更大比例的基础电路，代价是任务适应较慢。这些发现表明，电路保留可能有助于解释为什么RL对灾难性遗忘更具鲁棒性。我们在此发布了代码：https://github.com/rl-sft-circuit-research/differential-circuit-vulnerability。
英文摘要：
Fine-tuning large language models (LLMs) frequently induces catastrophic forgetting of prior capabilities. Recent work has shown that reinforcement learning (RL) retains prior capabilities more effectively than supervised fine-tuning (SFT), attributing this to policy-gradient updates remaining closer to the base policy \cite{shenfeld2025rl}. We extend this behavioral account to the mechanistic level and ask whether RL's advantage is mirrored by stronger preservation of internal computational circuits. We introduce differential circuit vulnerability, a head-level measure of how much a circuit degrades under fine-tuning, and use it to compare RL and SFT on Qwen2.5-3B-Instruct adapted to scientific question-answering. We find a clear mechanistic trade-off: SFT adapts more rapidly to the target task but produces substantially greater circuit disruption and forgetting of prior capabilities, whereas RL preserves a larger fraction of the base circuit at the cost of slower task adaptation. These findings suggest that circuit preservation may help explain why RL is more robust to catastrophic forgetting. We released our code here: https://github.com/rl-sft-circuit-research/differential-circuit-vulnerability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
