在固定预算下最大化 rollout 信息量：一种用于工具使用代理强化学习的子模性视角的树搜索
Maximizing Rollout Informativeness under a Fixed Budget: A Submodular View of Tree Search for Tool-Use Agentic Reinforcement Learning
AI总结：
本文提出在固定预算下最大化 rollout 信息量的问题，通过将中间状态选择转化为单调子模最大化问题，提出 InfoTree 框架，结合 UUCB 和 ABA，提升性能。
摘要：
我们正式将固定预算下的 rollout 信息量（RIFB）定义为工具使用 rollout 集对群体相对策略优化（GRPO）注入的预期非消失策略梯度质量。我们证明任何不依赖预算的独立采样器在硬提示下都会遭受有界远离零的崩溃率。受此启发，我们将中间状态选择转化为单调子模最大化问题，其中贪心一步选择器具有 1 减 1/e 的近似保证。我们的不确定性感知上置信界（UUCB）项作为该目标的闭式边际增益出现。这将 token 级熵奖金从经验技巧转变为该形式化的分析结果。我们提出了 InfoTree，一种训练时结合 UUCB 与学习的自适应预算分配器（ABA）和异步推测扩展方案的树搜索框架。ABA 挽救初始树被均匀结果浪费的提示，将混合结果比例从 58.1% 提高到 76.3%，以低于 5% 的预算开销。推测扩展通过容忍 UUCB 得分中的有界陈旧性，将时间开销从 14.3% 降至 4.8%。在九个涵盖数学推理（AIME 2024 和 2025，MATH-500，OlympiadBench，USAMO）、网络搜索代理（GAIA，HLE-100，BrowseComp-lite）以及工具丰富的编程和操作系统代理（APPS-verified，AgentBench-OS）的基准上，InfoTree 超过 flat GRPO、DeepSearch、Tree-GRPO、AT2PO、CW-GRPO 和 RC-GRPO。与 Tree-GRPO 前缀共享和 CW-GRPO 贡献权重的头对头组合进一步提升性能，确认我们的选择器在 rollout 重用和轨迹重加权方面是正交的。5x5x5 的鲁棒性网格显示，超四分之三的超参数空间位于性能平面上，确认 UUCB 的鲁棒性。
英文摘要：
We formalize Rollout Informativeness under a Fixed Budget (RIFB) as the expected non-vanishing policy-gradient mass that a tool-use rollout set injects into Group Relative Policy Optimization (GRPO). We prove that any budget-agnostic independent sampler suffers a collapse rate bounded away from zero for hard prompts regardless of the budget. Motivated by this, we recast intermediate state selection as a monotone submodular maximization problem, where a greedy one-step selector enjoys a 1 minus 1/e approximation guarantee. Our Uncertainty-aware Upper Confidence Bound (UUCB) terms arise as closed-form marginal gains of this objective. This turns the token-level entropy bonus from an empirical trick into an analytic consequence of the formulation. We present InfoTree, a training-time tree-search framework coupling UUCB with a learned Adaptive Budget Allocator (ABA) and an asynchronous Speculative Expansion scheme. ABA rescues prompts whose initial tree is wasted on uniform outcomes, lifting the mixed-outcome ratio from 58.1 percent to 76.3 percent with less than 5 percent budget overhead. Speculative Expansion reduces wall-clock overhead from 14.3 percent to 4.8 percent by tolerating bounded staleness in UUCB scores. Across nine benchmarks spanning math reasoning (AIME 2024 and 2025, MATH-500, OlympiadBench, USAMO), web-search agents (GAIA, HLE-100, BrowseComp-lite), and tool-rich coding and OS agents (APPS-verified, AgentBench-OS), InfoTree outperforms flat GRPO, DeepSearch, Tree-GRPO, AT2PO, CW-GRPO, and RC-GRPO. Head-to-head compositions with Tree-GRPO prefix sharing and CW-GRPO contribution weights deliver further gains, confirming that our selector operates orthogonally to rollout reuse and trajectory re-weighting. A 5 by 5 by 5 robustness grid reveals that over three quarters of the hyperparameter space lies on a performance plateau, confirming UUCB robustness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
