arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.07367cs.LG
收藏

自演化分布内优化的LLM智能体

Self-evolving LLM agents with in-distribution Optimization

University of Science and Technology of China(中国科学技术大学)

Yudi Zhang, Meng Fang, Zhenfang Chen, Mykola Pechenizkiy

AI总结:

提出Q-Evolve框架,通过分布内强化学习统一过程奖励标注与策略学习,利用加权隐式Q学习稳定贝尔曼更新,实现智能体自演化,在AlfWorld等任务上优于基线。

摘要:

大型语言模型(LLM)最近已成为复杂环境中交互智能体的强大控制器,但训练它们执行可靠的长期决策仍然是一个基本挑战。一个关键困难在于信用分配:智能体通常仅在回合结束时收到延迟奖励。在本文中,我们提出了Q-Evolve,一个用于LLM智能体的自演化框架,该框架在原则性的分布内强化学习范式中统一了自动过程奖励标注和策略学习。在每个演化迭代中,我们的方法从混合离策略数据集(结合专家演示与智能体生成的轨迹)中学习一个分布内评论家,通过加权隐式Q学习目标在稀疏奖励设置中稳定贝尔曼备份。然后,通过学习到的价值函数通过优势估计推导出逐步过程奖励,无需环境回溯或人工标注即可提供密集且可靠的监督。利用这些信号,我们执行行为近端策略优化,使智能体在用于过程奖励标注的数据上演化,从而在不加剧分布偏移的情况下实现迭代自我改进。我们在AlfWorld、WebShop和ScienceWorld上评估了我们的方法,结果显示Q-Evolve在样本效率、鲁棒性和整体任务性能上优于强基线。我们的结果表明,通过过程级监督和策略的共同演化(两者都基于共享的分布内学习循环),可以实现稳定的智能体自演化。

英文摘要:

Large Language Models (LLMs) have recently emerged as powerful controllers for interactive agents in complex environments, yet training them to perform reliable long-horizon decision making remains a fundamental challenge. A key difficulty lies in credit assignment: agents often receive delayed rewards only at the end of episodes. In this paper, we propose Q-Evolve, a self-evolving framework for LLM agents that unifies automatic process-reward labeling and policy learning within a principled in-distribution reinforcement learning paradigm. In each evolving iteration, our method learns an in-distribution critic from a hybrid off-policy dataset that combines expert demonstrations with agent-generated trajectories, stabilizing Bellman backups in sparse-reward settings via a weighted Implicit Q-Learning objective. The learned value function is then used to derive step-wise process rewards through advantage estimation, enabling dense and reliable supervision without environment backtracking or human annotation. Leveraging these signals, we perform behavior-proximal policy optimization that evolves the agent over the data used for process reward labeling, allowing iterative self-improvement without exacerbating distribution shift. We evaluate our method on AlfWorld, WebShop, and ScienceWorld, showing Q-Evolve outperforms strong baselines in sample efficiency, robustness, and overall task performance. Our results demonstrate that stable agent self-evolution is achievable through the co-evolution of process-level supervision and policy, both grounded within a shared in-distribution learning loop.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑