arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.08706cs.AIcs.CLcs.LG
收藏

代理批判训练

Agentic Critical Training

University of Maryland College Park(马里兰大学学院公园分校)

Weize Liu, Minghui Liu, Sy-Tuyen Ho, Yongkyun Lee, Andrew Adams Schoen, Souradip Chakraborty, Xiyao Wang, Furong Huang

AI总结:

代理批判训练(ACT)是一种强化学习方法,通过奖励模型判断替代行为的正确性,使模型自主发展对行为质量的推理,从而提升代理性能和泛化能力。

摘要:

训练大型语言模型(LLMs)作为自主代理往往从模仿学习开始,但这种方法只能教代理做什么而不能理解为什么:代理从不将成功的行为与次优替代行为进行对比,因此缺乏对行为质量的意识。最近的方法尝试通过引入来自专家和替代行为之间的对比所衍生的自我反思监督来解决这个问题。然而,训练范式本质上仍然是模仿学习:模型模仿预先构建的反思文本,而不是学习自主推理。我们提出代理批判训练(ACT),一种强化学习范式,训练代理识别替代行为中的更好行为。通过奖励模型判断是否正确,ACT促使模型自主发展对行为质量的推理,产生真正的自我反思,而不是模仿它。在三个具有挑战性的代理基准上,ACT结合不同的后训练方法时,始终能提升代理性能。它在模仿学习上平均提升5.07分,在强化学习上平均提升4.62分。与通过知识蒸馏注入反思能力的方法相比,ACT也显示出明显优势,平均提升2.42分。此外,ACT在代理基准上实现了强大的分布外泛化,并且在没有推理特定训练数据的情况下,提高了在一般推理基准上的性能,突显了我们方法的价值。这些结果表明,ACT是朝着开发更反思和有能力的LLM代理的一个有希望的途径。

英文摘要:

Imitation learning (IL) teaches language-model agents to reproduce expert actions but not to distinguish them from plausible mistakes. Self-reflection methods expose models to alternatives yet use supervised fine-tuning (SFT) to imitate fixed rationales and actions. We introduce Agentic Critical Training (ACT), which uses reinforcement learning with verifiable rewards (RLVR) to train models to judge actions directly. At each expert-trajectory state, ACT pairs an expert action with an alternative sampled from the initial policy and randomizes their order. The model generates its own reasoning but is rewarded only for selecting the expert action. ACT reuses demonstrations, requires no reference rationales, and allows pair reuse across model sizes. ACT is a warm-up before IL, optionally followed by RL; inference requires no candidate comparison. Across Qwen3-8B and Olmo-3-7B-Instruct on ALFWorld-ID, WebShop, and ScienceWorld, ACT yields average gains of 5.85 points over IL and 4.12 points over IL$\to$RL without ACT, while also improving ALFWorld-OOD. With Olmo on ScienceWorld, the full pipeline gains 15.36 points over CoT prompting and 9.23 points over IL$\to$RL without ACT. Both ACT$\to$IL and the full pipeline outperform supervised reflection baselines. Controls with fixed pairs or matched training durations show that the gains stem from the ACT objective rather than additional data or training. Without reasoning-specific post-training, the standalone ACT checkpoint achieves the highest mean among evaluated models on MATH-500 and GPQA-Diamond, showing that action comparison complements generation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑