Co-Evolution of Policy and Internal Reward for Language Agents
McGill University（麦吉尔大学）；McMaster University（麦克马斯特大学）；The University of Hong Kong（香港大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Peking University（北京大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；DeepWisdom（深度智慧）；Université de Montréal（蒙特利尔大学）；Mila（米拉研究所）
Xinyu Wang, Hanwei Wu, Jingwei Song, Shuyuan Zhang, Jiayi Zhang, Fanqi Kong, Tung Sum Thomas Kwok, Xiao-Wen Chang, Yuyu Luo, Chenglin Wu, Bang Liu
Large language model (LLM) agents learn by interacting with environments, but long-horizon training remains fundamentally bottlenecked by sparse and delayed rewards. Existing methods typically address this challenge through post-hoc credit assignment or external reward models, which provide limited guidance at inference time and often separate reward improvement from policy improvement. We propose Self-Guide, a self-generated internal reward for language agents that supports both inference-time guidance and training-time supervision. Specifically, the agent uses Self-Guide as a short self-guidance signal to steer the next action during inference, and converts the same signal into step-level internal reward for denser policy optimization during training. This creates a co-evolving loop: better policy produces better guidance, and better guidance further improves policy as internal reward. Across three agent benchmarks, inference-time self-guidance already yields clear gains, while jointly evolving policy and internal reward with GRPO brings further improvements (8\%) over baselines trained solely with environment reward. Overall, our results suggest that language agents can improve not only by collecting more experience, but also by learning to generate and refine their own internal reward during acting and learning.
SocraticPO: Policy Optimization via Interactive Guidance
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China（认知智能国家重点实验室，中国科学技术大学）；iFLYTEK AI Research (Central China), iFLYTEK Co., Ltd（iFLYTEK中央中国AI研究院，iFLYTEK公司）
PRPO: Aligning Process Reward with Outcome Reward in Policy Optimization
Shanghai University（上海大学）；University of Science and Technology of China（中国科学技术大学）；Shanghai Jiaotong University（上海交通大学）；Artificial Intelligence Incubation and Innovation Institute, Fudan University（复旦大学人工智能孵化与创新研究院）；Shanghai Academy of AI for Science（上海人工智能科学研究院）
RC-GRPO: Reward-Conditioned Group Relative Policy Optimization for Multi-Turn Tool Calling Agents
New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences（模式识别新实验室、多模态人工智能系统国家重点实验室、自动化研究所，中国科学院）；School of Mathematics and Statistics（数学与统计学学院）；Statistics, Lanzhou University（统计学，兰州大学）；Shanghai Innovation Institute（上海创新研究院）；Microsoft Research（微软研究院）；Nanjing University（南京大学）；Zhongguancun Academy（中关村学院）
Don't Mix Rewards, Mix Policies: Policy Decomposition and Optimization for Multi-Reward RL
Fundation Model Research Center, CASIA（中国科学院自动化研究所基础模型研究中心）；School of Artificial Intelligence, UCAS（中国科学院大学人工智能学院）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；College of Automotive and Energy Engineering (CAEE), Tongji University（同济大学汽车与能源工程学院）
奖励函数是强化学习（RL）中必不可少的，作为激励智能体解决给定任务的指导信号，然而，也是众所周知的难以设计。在许多情况下，只有不完美的奖励是可用的，这对RL代理造成了重大的性能损失。在这项研究中，我们提出了一个统一的离线策略优化方法，\textit{RGM (Reward Gap Minimization)}，它可以巧妙地处理不同类型的不完美奖励。RGM被制定为一个双层优化问题：上层优化的奖励校正项，执行访问分布匹配w.r.t.一些专家数据;较低层用校正的奖励解决悲观RL问题。通过利用较低层的二元性，我们得到了一个易于处理的算法，使基于采样的学习没有任何在线交互。综合实验表明，RGM实现了优越的性能，现有的方法在不同的设置不完美的奖励。此外，RGM可以有效地纠正错误的或不一致的奖励对专家的偏好和检索有用的信息，从有偏见的奖励。
Dynamics-Aware Comparison of Learned Reward Functions
Toyota Research Institute (TRI)（丰田研究院（TRI））；University of California, Berkeley（加州大学伯克利分校）
学习奖励函数的能力在现实世界中部署智能代理方面发挥着重要作用。然而，比较奖励函数，例如作为评估奖励学习方法的一种手段，提出了一个挑战。奖励函数通常通过考虑优化策略的行为来进行比较，但这种方法将奖励函数中的缺陷与用于优化它的策略搜索算法的缺陷混为一谈。为了解决这一挑战，Gleave et al.（2020）提出了等价策略不变比较（EPIC）距离。EPIC避免了策略优化，但这样做需要在转换时计算奖励值，这在系统动态下可能是不可能的。这对于学习的奖励函数来说是有问题的，因为它需要在训练分布之外对它们进行评估，导致不准确的奖励值，我们发现这会使EPIC在比较奖励时无效。为了解决这个问题，我们提出了动态感知奖励距离（DARD），一个新的奖励伪度量。DARD使用环境的近似转换模型将奖励函数转换为允许比较的形式，该比较对于奖励塑造是不变的，同时仅在接近其训练分布的转换上评估奖励函数。在模拟物理域的实验表明，DARD使可靠的奖励比较没有政策优化，是显着更具有预测性比基线方法的下游政策性能时，处理学习的奖励功能。
University at Buffalo, The State University of New York（纽约州立大学布法罗分校）；University of Colorado Boulder（科罗拉多大学博尔德分校）；University of Central Florida（中佛罗里达大学）；Air Force Research Laboratory（空军研究实验室）
