arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.25852cs.LGcs.AI
收藏

语义一致性策略优化:用于LLM智能体强化学习

Semantic Consistency Policy Optimization for Reinforcement Learning of LLM Agents

The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学)

Peng Xu, Sijia Chen, Junzhuo Li, Xuming Hu

AI总结:

针对基于组的强化学习中语义一致的中间步骤因轨迹成败获得相反信用的问题,提出无价值奖励塑造方法SCPO,通过从组内成功兄弟步骤恢复信用,在ALFWorld和WebShop上达到93.7%和74.8%的成功率。

摘要:

基于组的强化学习通过从轨迹结果推导步骤级信用,有效地对LLM智能体进行后训练,以处理长时域、稀疏奖励任务。然而,这将步骤的信用与其rollout的最终结果绑定:语义上几乎相同的中间步骤根据其轨迹最终成功或失败而获得相反的信用。这种语义信用不一致性向相似动作发送冲突梯度,并浪费了失败rollout中部分正确的进展。受此启发,我们提出语义一致性策略优化(SCPO),一种无价值的奖励塑造方法,通过从同一rollout组中的成功兄弟步骤恢复步骤级信用来缓解这种不一致性。具体地,SCPO将每个失败步骤与一个成功兄弟步骤进行评分,并为沿着该兄弟步骤的新进展添加正向步骤级信用。在ALFWorld和WebShop上,SCPO达到或超过强基线的基于组的方法,在1.5B参数下,ALFWorld上成功率为93.7±4.1%,WebShop上为74.8±2.0%,且增益集中在最困难的多步骤任务上。

英文摘要:

Group-based reinforcement learning effectively post-trains LLM agents for long-horizon, sparse-reward tasks by deriving step-level credit from trajectory outcomes. However, this ties a step's credit to its rollout's final outcome: semantically near-identical intermediate steps receive opposite credit depending on whether their trajectory eventually succeeded or failed. Such semantic credit inconsistency sends conflicting gradients to similar actions and wastes the partially-correct progress inside failed rollouts. Motivated by this, we propose Semantic Consistency Policy Optimization (SCPO), a value-free reward-shaping method that mitigates this inconsistency by recovering step-level credit from successful siblings in the same rollout group. Concretely, SCPO scores each failed step against a successful sibling and adds positive step-level credit for new progress along that sibling. On ALFWorld and WebShop, SCPO matches or exceeds strong group-based baselines, reaching 93.7+/-4.1 percent success on ALFWorld and 74.8+/-2.0 percent on WebShop at 1.5B parameters, with gains concentrated on the hardest multi-step tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑