TRIAGE：面向智能体强化学习的角色类型化信用分配
TRIAGE: Role-Typed Credit Assignment for Agentic Reinforcement Learning
AI总结：
提出TRIAGE框架，通过角色类型化信用分配修正GRPO仅依赖结果信用的盲点，在ALFWorld等任务中提升成功率并减少交互步数。
摘要：
智能体强化学习需要将信用分配给环境交互动作，如搜索、点击、编辑、导航命令和对象交互。标准GRPO使用最终验证器结果作为所有动作令牌的统一优势。该结果信号有用但结构不完整：它在失败轨迹中惩罚有用的探索，并在成功轨迹中强化冗余或倒退动作。我们提出TRIAGE，一种角色类型化信用分配框架，为结果信用添加语义角色轴。结构化评判器将每个片段分类为决定性进展、有用探索、无进展基础设施或倒退，固定角色条件规则将这些标签映射到有界片段级过程奖励。这保持了验证器结果作为优化方向的来源，同时纠正了仅结果信用的两个主要盲点。我们进一步证明，角色条件信用是从角色标签本身可表达的最优片段级修正——将每片段优势残差投影到角色变量上——因此当评判器可靠时，固定角色常数减少优势估计误差，并将其与低方差策略梯度联系起来。在ALFWorld、Search-QA和WebShop上，TRIAGE在两种策略模型上均优于GRPO的成功率，并优于标量评判器推导的过程奖励和结果监督共享骨干价值基线。消融实验表明，增益来自角色类型化而非仅仅添加密集奖励：成功轨迹内倒退的可靠检测是主要贡献者，而探索信用提供一致的次要增益；在完成的ALFWorld和WebShop轨迹上，TRIAGE相对于GRPO还额外减少了10.4%和14.8%的环境交互步数。
英文摘要：
Agentic reinforcement learning requires assigning credit to environment-facing actions such as searches, clicks, edits, navigation commands, and object interactions. Standard GRPO uses the final verifier outcome as a uniform advantage over all action tokens. This outcome signal is useful but structurally incomplete: it punishes useful exploration in failed rollouts and reinforces redundant or regressive actions in successful rollouts. We propose TRIAGE, a role-typed credit assignment framework that adds a semantic role axis to outcome credit. A structured judge classifies each segment as decisive progress, useful exploration, no-progress infrastructure, or regression, and a fixed role-conditioned rule maps these labels to bounded segment-level process rewards. This keeps verifier outcomes as the source of optimization direction while correcting the two main blind spots of outcome-only credit. We further show that the Bayes-optimal role-measurable correction is the L2 projection of the per-segment advantage residual onto the role variable, and that TRIAGE's fixed role constants approximate this projection, reducing advantage estimation error whenever the judge is reliable; we connect this to lower-variance policy gradients. Across ALFWorld, Search-QA, and WebShop, TRIAGE improves success rates over GRPO for two policy models and outperforms both a scalar judge-derived process reward and an outcome-supervised shared-backbone value baseline. Ablations show that the gain comes from role typing rather than merely adding dense rewards: reliable detection of regression inside successful trajectories is the dominant contributor, while exploration credit provides a consistent secondary gain; on completed ALFWorld and WebShop rollouts, TRIAGE also reduces environment-facing turns by an additional $10.4\%$ and $14.8\%$ relative to GRPO.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
