Large Language Models (LLMs) have recently emerged as powerful controllers for interactive agents in complex environments, yet training them to perform reliable long-horizon decision making remains a fundamental challenge. A key difficulty lies in credit assignment: agents often receive delayed rewards only at the end of episodes. In this paper, we propose Q-Evolve, a self-evolving framework for LLM agents that unifies automatic process-reward labeling and policy learning within a principled in-distribution reinforcement learning paradigm. In each evolving iteration, our method learns an in-distribution critic from a hybrid off-policy dataset that combines expert demonstrations with agent-generated trajectories, stabilizing Bellman backups in sparse-reward settings via a weighted Implicit Q-Learning objective. The learned value function is then used to derive step-wise process rewards through advantage estimation, enabling dense and reliable supervision without environment backtracking or human annotation. Leveraging these signals, we perform behavior-proximal policy optimization that evolves the agent over the data used for process reward labeling, allowing iterative self-improvement without exacerbating distribution shift. We evaluate our method on AlfWorld, WebShop, and ScienceWorld, showing Q-Evolve outperforms strong baselines in sample efficiency, robustness, and overall task performance. Our results demonstrate that stable agent self-evolution is achievable through the co-evolution of process-level supervision and policy, both grounded within a shared in-distribution learning loop.
UCAS-Terminus AI Lab, University of Chinese Academy of Sciences(中国科学院大学Terminus AI实验室,中国科学院大学);Centre for Artificial Intelligence and Robotics, HKISI-CAS(人工智能与机器人中心,中国科学院香港研究院);School of Computer Science and Engineering, Nanjing University of Science and Technology(南京理工大学计算机科学与工程学院)
University of Pennsylvania(宾夕法尼亚大学);Brown University(布朗大学);Nanyang Technological University(南洋理工大学);University of Notre Dame(圣母大学);George Mason University(乔治梅森大学)
AgentStream: How Well Do Self-Evolving LLM Agents Perform Under Streaming Tasks?
University of Chinese Academy of Sciences(中国科学院大学);Microsoft(微软公司);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Nanjing University(南京大学)
Self-Evolving Multi-Agent Network for Industrial IoT Predictive Maintenance
Department of Electronics Technology, Faculty of Electrical Engineering and Informatics, Budapest University of Technology and Economics(布达佩斯技术与经济大学电子技术系);DataScienceWorld;Department of Computer Science, The University of Alabama at Birmingham(阿拉巴马大学伯明翰分校计算机科学系)
Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agents
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);Renmin University of China(中国人民大学);Princeton University(普林斯顿大学);Hong Kong University of Science and Technology(香港科技大学);Fudan University(复旦大学)
MemSkill: Learning and Evolving Memory Skills for Self-Evolving Agents
Nanyang Technological University(南洋理工大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Illinois Chicago(伊利诺伊大学芝加哥分校);Tsinghua University(清华大学)
HealthFlow: A Self-Evolving AI Agent with Meta Planning for Autonomous Healthcare Research
Peking University(北京大学);The University of Hong Kong(香港大学);The Hong Kong University of Science and Technology(香港科技大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)