TRACE：一种用于高效智能体强化学习的统一展开预算分配框架
TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning
AI总结：
针对多轮智能体强化学习中奖励对比度不足的问题，提出TRACE框架，通过将每个ReAct式思考-行动-观察步骤建模为语义节点，在固定采样预算内将预算分配到提示根和中间前缀，增强奖励对比，提升策略更新信号。
摘要：
具有可验证奖励的强化学习（RLVR）是增强大型语言模型推理和智能体行为的一种有前景的方法。然而，展开密集的策略优化常常受到奖励对比度不足的限制，当过于简单或复杂的提示产生低方差反馈，以及当仅结果奖励对多轮展开中的每个决策赋予相同的终端评估时，就会出现这种情况。过去的努力集中在将可用的展开资源分配给有希望的提示，但它们仅利用提示级别的样本信息性，而忽略了同一展开中不同轮次之间前缀级别信息性的变化。本工作针对多轮智能体强化学习，将每个ReAct风格的思考-行动-观察步骤建模为语义上不同的节点，使得预算分配从提示根扩展到具有进一步延续的轮次级前缀，这自然形成了树状结构的展开。我们引入了树状展开分配用于对比探索（TRACE），这是一个统一的展开分配框架，在固定采样预算内增强奖励对比。在技术上，TRACE将展开预算分配给最可能产生混合终端奖励的提示根和中间前缀。一个共享的通用预测器根据前缀历史估计这些锚点处的条件成功概率，以指导这种分配。由此产生的自适应树状结构丰富了仅结果反馈，并放大了策略更新信号。实验上，TRACE在典型的智能体基准测试中取得了有竞争力的性能和效率提升，例如，在相同采样成本下，Qwen3-14B多跳问答的平均准确率比竞争基线提高了2.8个百分点。
英文摘要：
Reinforcement learning with verifiable rewards (RLVR) is a promising approach for enhancing reasoning and agentic behavior in large language models. However, rollout-intensive policy optimization is often limited by insufficient reward contrast, arising when overly simple or complex prompts generate low-variance feedback and when outcome-only rewards assign the same terminal assessment to every decision in a multi-turn rollout. Past efforts have focused on allocating available rollout resources to promising prompts, yet they only leverage sample informativeness at the prompt level and neglect variation in prefix-level informativeness across turns within the same rollout. This work targets multi-turn agentic RL by modeling each ReAct-style thought-action-observation turn as a semantically distinct node, allowing budget allocation to extend from prompt roots to turn-level prefixes with further continuations, which naturally forms tree-structured rollouts. We introduce Tree Rollout Allocation for Contrastive Exploration (TRACE), a unified rollout allocation framework that enhances reward contrast within a fixed sampling budget. Technically, TRACE allocates rollout budget to both prompt roots and intermediate prefixes that are most likely to yield mixed terminal rewards. A shared generalizable predictor estimates conditional success probability at these anchors from prefix histories to guide this allocation. The resulting adaptive tree structure enriches outcome-only feedback and amplifies the policy-update signal. Empirically, TRACE achieves competitive performance and efficiency gains on typical agentic benchmarks, e.g., improving Qwen3-14B Multi-Hop QA average accuracy by 2.8 points over competitive baselines at equal sampling cost.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
