Reinforcement learning (RL) holds the promise of learning a policy that can outperform what is shown in the data. However, in practice, such systems suffer from instability and hyperparameter sensitivity. In contrast, supervised learning provides a stable and scalable training objective, but is fundamentally limited by the quality of the demonstrated behavior. In this paper, we ask whether we can perform policy optimization with something as simple and stable as supervised learning but as performant as RL. Unfortunately, naively training the policy with supervised learning and conditioning on future rewards or advantages typically underperform dedicated RL methods. We introduce a new policy extraction method, Advantage Guidance, that retains the simple supervised learning structure with advantage conditioning while outperforming prior RL methods through effective conditional training and contrastive diffusion guidance during sampling. We show that Advantage Guidance can significantly improve a model's ability to respond to conditional advantage signals, and the contrastive guidance during sampling can use the learned advantage-conditioned policy to target the KL-regularized optimal policy. Empirically, we find that Advantage Guidance significantly outperforms prior state-of-the-art RL methods on 26 long-horizon, sparse reward tasks across datasets with varying degrees of optimality.
Step-by-Step Guidance to Differential Anemia Diagnosis with Real-World Data and Deep Reinforcement Learning
Inria Paris（法国国家信息与自动化研究所巴黎中心）；Centre de Recherche des Cordeliers（科尔德利耶研究所）；Inserm（法国国家健康与医学研究院）；Université Paris Cité（巴黎西岱大学）；Sorbonne Université（索邦大学）；Hôpital Européen Georges Pompidou（欧洲乔治·蓬皮杜医院）；Assistance Publique - Hôpitaux de Paris（巴黎公共医疗援助医院）；Hôpital Necker（内克尔医院）
Reinforcement Learning with Curriculum-inspired Adaptive Direct Policy Guidance for Truck Dispatching
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Safety-Regulated Transfer Reinforcement Learning with Adaptive Teacher Guidance
State Key Laboratory of Advanced Design and Manufacturing Technology for Vehicle, College of Mechanical and Vehicle Engineering, Hunan University（湖南大学机械与运载工程学院，先进设计与制造技术国家重点实验室）；Key Laboratory of Intelligent Air-Ground Cooperative Control for Universities in Chongqing, School of Automation, Chongqing University of Posts and Telecommunications（重庆邮电大学自动化学院，重庆市高校智能空地协同控制重点实验室）；School of Traffic and Transportation Engineering, Central South University（中南大学交通运输工程学院）
Connections between reinforcement learning with feedback,test-time scaling, and diffusion guidance: An anthology
Department of Statistics and Data Science, Chinese University of Hong Kong（统计与数据科学系，香港中文大学）；Department of Statistics and Data Science, the Wharton School, University of Pennsylvania（统计与数据科学系，沃顿商学院，宾夕法尼亚大学）
Action Guidance with MCTS for Deep Reinforcement Learning
Borealis AI
深度强化学习近年来取得了巨大的成功，但其面临的主要挑战是样本效率低下。在本文中，我们专注于如何使用行动指导的手段，非专家演示，以提高样本效率在一个域稀疏，延迟，并可能欺骗性的奖励：最近提出的多智能体基准的Pommerman。我们提出了一个新的框架，即使是一个非专家模拟演示，例如，诸如具有少量展开的蒙特卡罗树搜索的规划算法可以集成在异步分布式深度强化学习方法内。与vanilla deep RL算法相比，我们提出的方法在两人迷你版的Pommerman游戏中学习速度更快，收敛到更好的策略。
Efficient Multi-Task Reinforcement Learning with Cross-Task Policy Guidance
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Tsinghua University（清华大学）；Tencent AI Lab（腾讯AI实验室）
Diffusion Self-Weighted Guidance for Offline Reinforcement Learning
Initiative for Data & AI, Universidad de Chile（数据与人工智能倡议，智利大学）；AMTC & Dept. of Electrical Eng., Universidad de Chile（AMTC与电气工程系，智利大学）；Department of Mathematics, Imperial College London（数学系，伦敦帝国学院）
Where to go: Agent Guidance with Deep Reinforcement Learning in A City-Scale Online Ride-Hailing Service
Utah State University（犹他州立大学）
在线乘车服务已成为世界各地流行的交通系统。在本文中，我们研究了一个具有挑战性的问题，即如何在城市周围引导空置的出租车，以便在在线乘车服务中实现供需平衡。我们设计了一个新的奖励计划，考虑了在线乘车服务的多个性能指标。我们还提出了一种新的深度强化学习方法，名为Deep-Q-Network with Action Mask（AM-DQN），该方法屏蔽了各个位置的不必要动作，从而使智能体可以更快，更有效地学习。我们使用芝加哥的城市规模数据集进行了广泛的实验。几种流行的启发式和学习方法也被实现为比较的基线。实验结果表明，AM-DQN的平均故障率，平均等待时间的客户，平均空闲搜索时间的空闲出租车方面达到所有方法的最佳性能。
