面向进度和可靠性的智能体强化学习组策略优化
Progress- and Reliability-Oriented Group Policy Optimization for Agentic Reinforcement Learning
AI总结：
研究基于组的强化学习，针对步级优势估计对组形成方式敏感的问题，提出ProGPO方法，通过精确前缀动作比较及结合语义扩展与逆方差融合估计势，在任务中改进了基线方法。
摘要：
基于组的强化学习已成为改进大语言模型智能体在长期交互任务上的有效范式。近期工作转向步级基于组的强化学习，但步级优势估计对组的形成方式敏感。本文提出ProGPO，一种无学习评论家的上下文一致步级学习方法，通过精确前缀动作比较及结合语义扩展与逆方差融合估计势，在任务中改进了基线方法。
英文摘要：
Group-based reinforcement learning (RL) has become an effective paradigm for improving large language model agents on long-horizon interactive tasks. To obtain finer-grained policy updates than trajectory-level optimization, recent work has moved toward step-level group-based RL, where intermediate steps are grouped and compared within a rollout batch. However, step-level advantage estimation is sensitive to how groups are formed: grouping by broad state keys improves coverage but may compare actions taken under different histories, while enforcing historical consistency yields fairer comparisons at the cost of fragmented groups and missing peer-comparison signal. In this paper, we propose ProGPO (Progress- and Reliability-Oriented Group Policy Optimization), a learned-critic-free method for context-consistent step-level learning. ProGPO keeps exact-prefix action comparison, and complements sparse peer comparisons with transition credit derived from rollout-based state potentials. To estimate these potentials reliably, ProGPO combines semantic expansion with inverse-variance fusion across history depths. We evaluate ProGPO on two challenging agentic tasks, ALFWorld and WebShop, with Qwen2.5-1.5B-Instruct. Results show that ProGPO improves over matched agentic RL baselines under comparable computational overhead, and additional Qwen2.5-3B-Instruct experiments further test the scalability of the proposed method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
