AdaStep：智能体强化学习中的自适应步骤信用加权
AdaStep: Adaptive Step Credit Weighting for Agentic Reinforcement Learning
浏览论文内容
中文总结 AI 辅助
针对长视界LLM智能体稀疏奖励下信用分配不精确的问题，提出AdaStep自适应步骤信用加权方法，通过最优收缩系数平衡局部与全局信号，在低计算成本下提升多环境性能。
中文摘要 AI 辅助
长视界LLM智能体通常使用稀疏的结果奖励进行训练，这使得轨迹级目标过于粗糙，难以区分单个决策的贡献。步骤级信用分配提供了更细粒度的监督，但其估计可能不可靠，因为观察到的回报也取决于后续动作、环境转换和轨迹长度。我们提出AdaStep，一种自适应步骤信用加权方法，控制每个组派生的局部优势对轨迹级信号的调节强度。我们将此加权问题表述为潜在步骤优势的均方误差估计问题，并在显式条件抽样假设下，推导出最优的每状态收缩系数。该系数具有信号与总方差比的解释：当回报变化可归因于所选动作时，保留局部信用；当变化由下游随机性主导时，抑制局部信用。AdaStep仅需轻量级标量计算，无需评论家、额外轨迹或额外模型推理。在ALFWorld、WebShop和ScienceWorld上使用三种模型骨干进行的实验显示，在低计算成本下相对于基线的一致改进。
英文摘要
Long-horizon LLM agents are typically trained with sparse outcome rewards, making trajectory-level objectives too coarse to distinguish the contribution of individual decisions. Step-level credit assignment provides finer-grained supervision, but its estimates can be unreliable because observed returns also depend on subsequent actions, environment transitions, and trajectory length. We propose AdaStep, an Adaptive Step-credit weighting method that controls how strongly each group-derived local advantage modifies the trajectory-level signal. We formulate this weighting as a mean-squared-error estimation problem for the latent step advantage and, under an explicit conditional sampling assumption, derive an optimal per-state shrinkage coefficient. The coefficient admits a signal-to-total-variance interpretation: it preserves local credit when return variation is attributable to the selected action and suppresses it when variation is dominated by downstream randomness. AdaStep requires only lightweight scalar computation, with no critic, additional rollouts, or extra model inference. Experiments with three model backbones on ALFWorld, WebShop, and ScienceWorld show consistent improvements over baselines at low computational cost.
发表机构
- Tsinghua University（清华大学）
- Nanjing University（南京大学）
- Xiaomi Inc.（小米公司）
机构由 AI 辅助整理，请以论文原文为准。
