arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.13755cs.LGcs.AI
收藏

用于大语言模型智能体训练的匹配上下文组策略优化

Matched-Context Group Policy Optimization for LLM Agent Training

Imperial College London(帝国理工学院);University of Science and Technology of China(中国科学技术大学);Meituan(美团);Tsinghua University(清华大学)

Chenxu Zhang, Yaorui Shi, Wenlong Zhang, Jianchao Tan, Zihan Wang, Yiming Cheng

AI总结:

该研究针对长视野大语言模型智能体训练的信用分配问题,提出MCGPO方法,通过匹配上下文构建加权动作偏好,在多环境多模型上提升了平均成功率。

摘要:

训练长视野大语言模型智能体需要将信用分配给稀疏终端反馈中的中间决策。在有限的一组并行 rollout 中,有用的动作比较依赖于它们的上下文、两种备选方案的证据,以及该证据如何进入学习过程。我们提出匹配上下文组策略优化(Matched-Context Group Policy Optimization,MCGPO),这是一种从已收集的轨迹中构建加权动作偏好的方法。MCGPO 用动作前状态摘要和任务规则标注的观察 rubric 进度定义比较上下文,双动作支持在有序精确匹配规则和粗匹配规则中进行选择。固定折将候选决策与用于确定其偏好的终端结果分开,依赖支持的证据权重与共享 GraphGPO 任务学习一起缩放辅助成对目标。MCGPO 复用 rollout 数据和每个响应的原始历史,保留无 critic 的训练。在六个环境-主干设置(每个设置使用三个训练种子)中,MCGPO 相对于匹配的 GraphGPO 提高了平均成功率:在 Qwen2.5-1.5B 和 7B 上的 ALFWorld 和 WebShop 上提高了 0.47-1.46 个百分点,在 Qwen3-4B 和 8B 上的 WebShop 上分别提高了 3.96 和 5.16 个百分点。ALFWorld 的增益建立在超过 92% 的基线成功率之上,剩余提升空间有限。在四个 Qwen2.5 设置中的三个中,综合成功率有所提高。三种子集的析因控制支持相反折的证据和证据加权;联合状态-rubric 匹配在所有四个表示控制设置中实现了最高平均值。

英文摘要:

Training long-horizon language model agents requires assigning credit to intermediate decisions from sparse terminal feedback. Within a finite group of sibling rollouts, useful action comparisons depend on their context, evidence for both alternatives, and how that evidence enters learning. We propose Matched-Context Group Policy Optimization (MCGPO), a procedure for constructing weighted action preferences from already collected trajectories. MCGPO defines comparison contexts with pre-action state summaries and observed rubric progress, annotated by task rules. Two-action support selects among ordered exact and coarse matching rules. Fixed folds separate candidate decisions from the terminal outcomes used to orient their preferences, and support-dependent evidence weights scale an auxiliary pairwise objective alongside shared GraphGPO task learning. MCGPO reuses rollout data and each response's original history, preserving critic-free training. Across six environment-backbone settings with three training seeds each, MCGPO improves mean success over matched GraphGPO: by 0.47-1.46 percentage points on ALFWorld and WebShop with Qwen2.5-1.5B and 7B, and by 3.96 and 5.16 points on WebShop with Qwen3-4B and 8B. The ALFWorld gains build on baseline success above 92%, where remaining headroom is limited. Integrated success improves in three of four Qwen2.5 settings. Three-seed factorial controls support opposite-fold evidence and evidence weighting; joint state-rubric matching achieves the highest mean in all four representation-control settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑