Reinforcement learning (RL) has emerged as the predominant paradigm for training large language model (LLM)-based AI agents. However, existing backbone RL algorithms lack verified convergence guarantees in agentic scenarios, especially in multi-turn settings, which can lead to training instability and failure to converge to optimal policies.
In this paper, we systematically analyze how different combinations of policy update mechanisms and advantage estimation methods affect convergence properties in single/multi-turn scenarios. We find that REINFORCE with Group Relative Advantage Estimation (GRAE) can converge to the globally optimal under undiscounted conditions, but the combination of PPO & GRAE breaks PPO's original monotonic improvement property. Furthermore, we demonstrate that mainstream backbone RL algorithms cannot simultaneously achieve both critic-free and convergence guarantees in multi-turn scenarios.
To address this, we propose SeeUPO (Sequence-level Sequential Update Policy Optimization), a critic-free approach with convergence guarantees for multi-turn interactions. SeeUPO models multi-turn interaction as sequentially executed multi-agent bandit problems. Through turn-by-turn sequential policy updates in reverse execution order, it ensures monotonic improvement and convergence to global optimal solution via backward induction.
Experiments on AppWorld and BFCL v4 demonstrate SeeUPO's substantial improvements over existing backbone algorithms: relative gains of 43.3%-54.6% on Qwen3-14B and 24.1%-41.9% on Qwen2.5-14B (averaged across benchmarks), along with superior training stability.
Limits to Verification and Validation of Agentic Behavior
代理行为的验证和确认已被建议作为重要的研究重点,以减少与创建通用人工智能相关的风险(Russell et al 2015)。在本文中,我们质疑使用语言的确定性方面的努力,以管理这种风险的适当性。我们首先建立了一个非常一般的形式主义来描述代理行为和可接受的行为的标准。我们表明,确定一个代理是否符合任何特定的标准是不可计算的。我们讨论了与人工证明和自动化行为治理验证相关的负担程度。我们表明,为了确保行为标准本身的可判定性,必须进一步限制代理的能力。然后,我们证明,如果我们的关注涉及到在物理世界的结果,验证的尝试是徒劳的。最后,我们表明,分层架构旨在使这些挑战易于处理的错误等同于意图与行动或结果,从而无法提供任何保证。最后,我们讨论了为什么确定性语言应该从关于一般人工智能安全性的对话中根除。
Optimizing Agentic Reasoning with Retrieval via Synthetic Semantic Information Gain Reward
Hong Kong JC STEM Lab of Smart City.(香港JC STEM实验室);City University of Hong Kong.(香港城市大学);Lingnan University(岭南大学);Fudan University.(复旦大学);Huazhong University of Science and Technology.(华中科技大学)
LLMZ+: Contextual Prompt Whitelist Principles for Agentic LLMs
Department of Computer Science Mississippi State University(计算机科学系密苏里州立大学);Department of Computer Science The University of Alabama(计算机科学系阿拉巴马大学);Mississippi State University, Mississippi State, MS, USA(密苏里州立大学);The University of Alabama, Tuscaloosa, AL, USA(阿拉巴马大学)
RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training
Adelaide University(阿德莱德大学);The University of Sydney(悉尼大学);Curtin University(科廷大学);School of CSIT(计算机科学与信息技术学院);ACFR(澳大利亚空间研究中心);School of EECMS(电子工程、计算机与数学科学学院)
SQL-ASTRA: Alleviating Sparse Feedback in Agentic SQL via Column-Set Matching and Trajectory Aggregation
Griffith University(格里菲斯大学);Fudan University(复旦大学);The University of Hong Kong(香港大学);Peking University(北京大学);InFly;Shanghai Innovation Institute(上海创新研究院);Shanghai Academy of Artificial Intelligence for Science(上海人工智能科学研究院)
From Privacy to Trust in the Agentic Era: A Taxonomy of Challenges in Trustworthy Federated Learning Through the Lens of Trust Report 2.0
Department of Computer Science and Artificial Intelligence, Andalusian Research Institute in Data Science and Computational Intelligence (DaSCI) University of Granada(计算机科学与人工智能系,数据科学与计算智能安达卢西亚研究 institute,格拉纳达大学);Department of Software Engineering, Andalusian Research Institute in Data Science and Computational Intelligence (DaSCI) University of Granada(软件工程系,数据科学与计算智能安达卢西亚研究 institute,格拉纳达大学)
A Blockchain-Monitored Agentic AI Architecture for Trusted Perception-Reasoning-Action Pipelines
Faculty of Computer Studies, Arab Open University-Bahrain(巴林阿拉伯开放大学计算机科学学院);Faculty of Computer and Information System, Islamic University of Madinah, Saudi Arabia(沙特阿拉伯麦地那伊斯兰大学计算机与信息系统学院)
FastOMOP: A Foundational Architecture for Reliable Agentic Real-World Evidence Generation on OMOP CDM data
Lancashire Teaching Hospitals NHS Foundation Trust(兰开夏教学医院国家健康服务信托基金会);Lancaster University(兰卡斯特大学);EPSRC DRIVE-Health, Department of Biostatistics & Health Informatics(EPSRC DRIVE-Health,生物统计学与健康信息学部门);King’s College London(伦敦国王学院);Institute for Health Informatics(健康信息学研究所);University College London(伦敦大学学院);NIHR Biomedical Research Centre(英国国家健康服务研究院生物医学研究中心);University College London Hospitals NHS Foundation Trust(伦敦大学学院医院国家健康服务信托基金会);Health Data Research UK London(英国健康数据研究伦敦);South London and Maudsley NHS Foundation Trust and King’s College London(伦敦南部及莫德利国家健康服务信托基金会和伦敦国王学院);Department of Biostatistics & Health Informatics, Institute of Psychiatry, Psychology & Neuroscience(生物统计学与健康信息学部门,精神病学、心理学与神经科学研究所)
LumiVideo: An Intelligent Agentic System for Video Color Grading
Northwestern University(西北大学);Northeastern University(东北大学);South China University of Technology(华南理工大学);Hong Kong Baptist University(香港浸会大学);Beijing Normal - Hong Kong Baptist University(北京师范大学-香港浸会大学)
Agentic-V2X: Small Language Model Agents for Deadline-Aware V2X Scheduling in 5G/6G Networks
Department of Informatics, University of Piraeus(比雷埃克斯大学信息学院);Department of Informatics and Telecommunications, University of Peloponnese(希腊佩拉索斯大学信息与电信学院)