RTMC：通过展开树进行步骤级信用分配
RTMC: Step-Level Credit Assignment via Rollout Trees
AI总结：
RTMC通过展开树方法实现步骤级信用分配，无需学习型批评者，提升SWE-bench Verified任务的pass@1性能3.2个百分点。
摘要：
多步代理强化学习受益于细粒度信用分配，但现有方法选项有限：无批评者方法如GRPO对轨迹中每个动作分配统一优势，而学习价值网络引入显著开销且在稀疏奖励下易脆弱。我们观察到针对同一问题的组展开往往遍历重叠的中间状态，隐式形成一个分支在后续决策点分歧的树。基于此见解，我们引入展开树蒙特卡洛（RTMC）优势估计，通过共享相同状态的展开聚合回报统计，生成每步Q值和优势——无需任何学习型批评者。一个状态-动作签名系统将原始交互历史压缩成紧凑、可比的表示，使跨展开状态匹配可行。在SWE-bench Verified上，RTMC比GRPO提升pass@1性能3.2个百分点。
英文摘要：
Multi-step agentic reinforcement learning benefits from fine-grained credit assignment, yet existing approaches offer limited options: critic-free methods like GRPO assign a uniform advantage to every action in a trajectory, while learned value networks introduce notable overhead and can be fragile under sparse rewards. We observe that group rollouts targeting the same problem often traverse overlapping intermediate states, implicitly forming a tree whose branches diverge at successive decision points. Building on this insight, we introduce Rollout-Tree Monte Carlo (RTMC) advantage estimation, which aggregates return statistics across rollouts sharing a common state to produce per-step Q-values and advantages--without any learned critic. A state-action signature system compresses raw interaction histories into compact, comparable representations, making cross-rollout state matching tractable. On SWE-bench Verified, RTMC improves pass@1 by 3.2 percentage points over GRPO.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
