推理模型差分的转码适配器
Transcoder Adapters for Reasoning-Model Diffing
AI总结:
转码适配器用于分析推理模型微调前后的差异,揭示了推理训练对模型内部机制的影响,并展示了适配器在恢复推理准确性方面的有效性。
摘要:
尽管推理模型日益普及,但推理训练对模型内部机制的影响仍不清晰。在本工作中,我们引入了转码适配器,这是一种学习MLP计算在微调前后差异的可解释近似方法。我们将转码适配器应用于刻画Qwen2.5-Math-7B与其推理蒸馏变体DeepSeek-R1-Distill-Qwen-7B之间的差异。学习到的适配器忠实于目标模型的内部计算和下一个词预测。在推理基准测试中,适配器匹配推理模型的响应长度,并通常恢复50-90%的推理微调准确性提升。适配器特征是稀疏激活且可解释的。当检查适配器特征时,我们发现只有约8%的特征直接与推理行为相关。我们深入研究了一种此类行为——产生犹豫标记(例如
英文摘要:
While reasoning models are increasingly ubiquitous, the effects of reasoning training on a model's internal mechanisms remain poorly understood. In this work, we introduce transcoder adapters, a technique for learning an interpretable approximation of the difference in MLP computation before and after fine-tuning. We apply transcoder adapters to characterize the differences between Qwen2.5-Math-7B and its reasoning-distilled variant, DeepSeek-R1-Distill-Qwen-7B. Learned adapters are faithful to the target model's internal computation and next-token predictions. When evaluated on reasoning benchmarks, adapters match the reasoning model's response lengths and typically recover 50-90% of the accuracy gains from reasoning fine-tuning. Adapter features are sparsely activating and interpretable. When examining adapter features, we find that only ~8% have activating examples directly related to reasoning behaviors. We deeply study one such behavior -- the production of hesitation tokens (e.g., "wait"). Using attribution graphs, we trace hesitation to only ~2.4% of adapter features (5.6k total) performing one of two functions. These features are necessary and sufficient for producing hesitation tokens; removing them reduces response length, often without affecting accuracy. Overall, our results provide insight into reasoning training and suggest transcoder adapters may be useful for studying fine-tuning more broadly.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。