Proximal Policy Optimization (PPO) is widely used in continuous control due to its robustness and stable training, yet it remains sample-inefficient in tasks with expensive interactions and high-dimensional action spaces. This paper proposes PPO-DAP (PPO with Diffusion Action Prior), a strictly on-policy framework that improves exploration quality and learning efficiency without modifying the PPO objective. PPO-DAP follows a two-stage protocol. Offline, we pretrain a conditional diffusion action prior on logged trajectories to cover the action distribution supported by the behavior policy. Online, PPO updates the actor-critic only using newly collected on-policy rollouts, while the diffusion prior is adapted around the on-policy state distribution via parameter-efficient tuning (Adapter/LoRA) over a small parameter subset. For each on-policy state, the prior generates multiple action proposals and concentrates them toward high-value regions using critic-based energy reweighting and in-denoising gradient guidance. These proposals affect the actor only through a low-weight imitation loss and an optional soft KL regularizer to the prior; importantly, PPO gradients are never backpropagated through offline logs or purely synthetic trajectories. We further analyze the method from a dual-proximal perspective and derive a one-step performance lower bound. Across eight MuJoCo continuous-control tasks under a unified online budget of 1.0M environment steps, PPO-DAP consistently improves early learning efficiency (area under the learning curve over the first 40 epochs, ALC@40) and matches or exceeds the strongest on-policy baselines in final return on 6/8 tasks, with modest overhead (1.18+/-0.04x wall-clock time and 1.05+/-0.02x peak GPU memory relative to PPO).
PPO-UE: Proximal Policy Optimization via Uncertainty-Aware Exploration
Virginia Tech(弗吉尼亚理工大学);University of Oslo(奥斯陆大学);DEVCOM Army Research Laboratory(DEVCOM陆军研究实验室);University of Texas at Dallas(德克萨斯大学达拉斯分校);University of the District of Columbia(哥伦比亚特区大学)
HEALing Entropy Collapse: Enhancing Exploration in Few-Shot RLVR via Hybrid-Domain Entropy Dynamics Alignment
School of Informatics, Xiamen University(厦门大学信息学院);Tsinghua University(清华大学);Xiamen Unisound Intelligence Technology Co., Ltd(厦门Unisound智能科技有限公司);Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China(福建省和台湾非物质文化遗产数字化保护与智能处理重点实验室(厦门大学),中华人民共和国文化和旅游部,中国)
Enhancing Efficiency and Exploration in Reinforcement Learning for LLMs
School of Computer Science and Technology, Beijing Jiaotong University(计算机科学与技术学院,北京交通大学);Meituan(美团);Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence(北京交通数据挖掘与具身智能重点实验室)
Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
School of Electrical;Archimedes, Athena Research Center, Greece;Department of Mathematics, ETH Zurich;Dept. of Mechanical Engineering \& Materials Science, Duke University;Dept. of Digital Systems, University of Piraeus, Greece
Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithm
Indian Institute of Science(印度科学学院)
在大多数强化学习(RL)算法的初始训练迭代期间,代理执行大量随机探索步骤。在现实世界中,这可能会限制这些算法的实用性,因为它可能导致潜在的危险行为。因此,安全探索是在现实世界中应用RL算法的一个关键问题。这个问题最近在约束马尔可夫决策过程(CMDP)框架下得到了很好的研究,其中除了单阶段奖励外,代理还根据状态转换接收单阶段成本或惩罚。规定的成本函数负责将任何给定时间步的不良行为映射到标量值。然后,目标是找到一个可行的政策,最大限度地提高回报,同时限制成本回报低于规定的阈值,在训练和部署。我们提出了一种基于策略模型的安全深度RL算法,在该算法中,我们以在线方式学习环境的过渡动态,并使用基于拉格朗日松弛的近端策略优化找到可行的最优策略。我们使用不同初始化的神经网络的集成来解决环境模型学习过程中所面临的认知和任意不确定性问题。我们使用具有挑战性的安全强化学习基准-Open AI Safety Gym,将我们的方法与约束RL中的相关无模型和基于模型的方法进行比较。我们证明,我们的算法是更有效的样本,并在较低的累积危险违规的结果相比,受约束的无模型的方法。此外,我们的方法显示出更好的回报性能比其他约束的基于模型的方法在文献中。
DGRO: Enhancing LLM Reasoning via Exploration-Exploitation Control and Reward Variance Management
School of Mathematics and Statistics(数学与统计学学院);Beijing Jiaotong University(北京交通大学);Yau Mathematical Sciences Center(叶中科学中心);Tsinghua University(清华大学);Department of Mathematical Sciences(数学科学系);Beijing, China(中国北京);Zhongguancun Academy(中关村学院);Yanqi Lake Beijing Institute of Mathematical Sciences and Applications(燕琪湖数学科学与应用研究所);Academy of Mathematics and Systems Science(数学与系统科学研究院);Chinese Academy of Sciences(中国科学院);Microsoft Research AI4Science(微软研究院AI4Science)
Colored Noise in PPO: Improved Exploration and Performance through Correlated Action Sampling
Department of Computer Science, University of Innsbruck(因斯布鲁克大学计算机科学系);Digital Science Center, University of Innsbruck(因斯布鲁克大学数字科学中心);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);Department of Computer Science, University of Tübingen(蒂宾根大学计算机科学系)
Enhancing Autonomous Vehicle Training with Language Model Integration and Critical Scenario Generation
Imperial College London(帝国理工学院);Centre for Transport Engineering and Modelling(交通工程与建模中心);Department of Civil and Environmental Engineering(土木与环境工程系);Personal Robotics Laboratory(个人机器人实验室)
EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Framework
Zhongguancun Academy(中关村学院);College of Software, Nankai University(南开大学软件学院);School of Computer Science, Shanghai Jiao Tong University(上海交通大学计算机科学学院);Academy of Mathematics and Systems Science, Chinese Academy of Sciences(中国科学院数学与系统科学研究院);Department of Electronic Engineering, Tsinghua University(清华大学电子工程系)
Optimistic ε-Greedy Exploration for Cooperative Multi-Agent Reinforcement Learning
University of Electronic Science and Technology of China(电子科技大学);Southwest Minzu University(西南民族大学);Chengdu University of Technology(成都理工大学);Nanyang Technological University(南洋理工大学)
Boosting Efficiency in Task-Agnostic Exploration through Causal Knowledge
Department of Computer Science and Engineering, Shanghai Jiao Tong University(上海交通大学计算机科学与工程系);Halicioğlu Data Science Institute (HDSI), University of California San Diego(加州大学圣迭戈分校哈利乔格鲁数据科学研究所);Guangdong Institute of Intelligence Science and Technology(广东省智能科学与技术研究院)