CRAFT：基于自由兄弟轨迹的反事实信用分配用于自蒸馏智能体强化学习
CRAFT: Counterfactual Credit Assignment from Free Sibling Rollouts for Self-Distilled Agentic Reinforcement Learning
AI总结：
提出CRAFT方法，通过反事实令牌重要性、非对称控制器和极化KL惩罚三个支柱，解决自蒸馏强化学习中信用分配信号的双重局限性，提升智能体性能。
摘要：
自蒸馏智能体强化学习通过令牌级蒸馏损失增强轨迹级奖励，其教师是同一策略但以特权上下文为条件。当前方法用单个标量（教师-学生对数概率差距）控制该损失，该信号有双重局限：它是回顾性的，只评分实际轨迹而非反事实轨迹；且是符号盲的，从不指示教师偏好的动作何时会损害轨迹。我们提出CRAFT，一种三支柱信用分配方案，解决这两个局限。支柱1，反事实令牌重要性，重用GRPO已采样的G-1个兄弟轨迹，并用对数概率差距进行重要性加权，形成组级反事实优势变化的自归一化估计，从而以近零额外计算获得每个令牌的有符号信用。支柱2是非对称控制器，根据门控活动的指数移动平均提高蒸馏权重同时降低参考KL权重，反之亦然。支柱3逐令牌极化KL惩罚，根据信用的符号在模式搜索和模式覆盖更新之间切换。每个支柱有独立开关，禁用时损失和梯度在IEEE-754算术上与基线字节相同，因此任何测量增益归因于算法变化而非实现漂移。我们证明了估计量的一致性和方差界，给出了结构和位精确可重复性保证，并在三个智能体环境、四个模型规模和五种端到端方法以及两个表格化的先前工作基线上评估了CRAFT。其中包括Adaptive-CRINGE，一个与CRAFT共享支柱2的比较器，隔离了反事实贡献。
英文摘要：
Self-distilled agentic reinforcement learning augments trajectory-level reward with a token-level distillation loss, using as its teacher the same policy conditioned on privileged context. The prevailing recipe gates this loss by a single scalar, the teacher-student log-probability gap. This signal is doubly limited: it is retrospective, scoring only the realised rollout and never the counterfactual ones, and it is sign-blind, never signalling when a teacher-preferred action would have harmed the trajectory. We introduce CRAFT, a three-pillar credit-assignment scheme that addresses both limitations. Pillar 1, Counterfactual Token Importance, reuses the G-1 sibling rollouts that GRPO already samples and importance-weights them by the log-probability gap to form a self-normalised estimate of the group-level counterfactual change in advantage from up-weighting teacher-preferred actions at each step; this yields a signed per-token credit at near-zero extra compute. Pillar 2 is an asymmetric controller that raises the distillation weight as it lowers the reference-KL weight along an exponential moving average of gate activity, and conversely. Pillar 3 polarises the KL penalty token by token, switching between a mode-seeking and a mode-covering update according to the sign of the credit. Each pillar has an independent switch that, when disabled, renders the loss and gradient byte-identical to the baseline in IEEE-754 arithmetic, so any measured gain is attributable to algorithmic change rather than implementation drift. We prove the estimator's consistency and a variance bound, give structural and bit-exact reproducibility guarantees, and evaluate CRAFT across three agentic environments, four model scales, and five end-to-end methods, plus two tabulated prior-work baselines. Among these is Adaptive-CRINGE, a comparator sharing Pillar 2 with CRAFT, isolating the counterfactual contribution.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
