USA：用于语言智能体跨域在线策略蒸馏的更新感知SAM

中文摘要 AI 辅助

在线策略蒸馏通过在学生自身的轨迹上提供密集的令牌级监督来灌输多轮智能体推理能力，但单一领域会过早饱和，因此进一步的监督必须从其他领域获取。多领域数据混合是纳入这些监督的最直接方式，但其代价是领域间数据分布冲突，以及每当一个领域被修订时都需要重新训练整个模型。模型合并通过独立蒸馏每个领域并随后融合所得的任务向量来避免这两种代价。然而，我们发现其收益在领域对之间呈现两极分化：在表现出负迁移的领域对上，我们评估的每个合并算子都低于单领域基准。我们将此归因于跨域更新耦合，即相当大比例的坐标被两个领域以可比幅度更新，因此合并可能使这些坐标的偏移量与其自身更新幅度相当。为克服这一局限，我们提出USA，该方法将在短暂预热期间测得的每参数更新幅度转换为每坐标扰动半径，从而精确地在承载大部分合并位移的坐标上降低曲率。在数学、科学和代码领域以及两种学生规模上的实验表明，USA在所有六个迁移方向上表现最强，平均领先单领域基准超过四个百分点，并逆转了冲突对的负迁移。

英文摘要

On-policy distillation instils multi-turn agentic reasoning through dense token-level supervision on the student's own trajectories, but a single domain saturates early, so further supervision has to be drawn from other domains. Multi-domain data mixing is the most direct way of incorporating them, at the cost of conflicts between their data distributions and of retraining the entire model whenever one domain is revised. Model merging avoids both by distilling every domain independently and fusing the resulting task vectors afterwards. We find instead that the benefit polarizes across domain pairs: on those exhibiting negative transfer, every merging operator we evaluate falls below the single-domain reference. We attribute this to cross-domain update coupling, where a substantial fraction of coordinates is updated comparably by both domains and a merge can therefore displace them by as much as their own updates. To overcome this limitation, we propose USA, which converts per-parameter update magnitudes measured during a brief warm-up into per-coordinate perturbation radii, reducing curvature precisely on the coordinates that carry most of the merging displacement. Experiments across mathematics, science and code at two student scales show USA strongest in all six transfer directions, ahead of the single-domain reference by more than four points on average, and reverse the negative transfer of the conflicting pairs.

发表机构

• University of Science and Technology of China（中国科学技术大学）
• Tencent（腾讯）
• Zhejiang University（浙江大学）
• National University of Singapore（新加坡国立大学）

机构由 AI 辅助整理，请以论文原文为准。