本文提出 EBFlow 框架，用基于能量的归一化流统一 MaxEnt RL 的策略评估与改进，实现无 Monte Carlo 的 soft 价值计算、多模态动作建模和高效采样，并在 MuJoCo 与 Isaac Gym 任务中优于基线。
摘要：
现有的面向连续动作空间的最大熵（Maximum-Entropy，MaxEnt）强化学习（RL）方法通常基于 actor-critic 框架构建，并通过策略评估与策略改进的交替步骤进行优化。在策略评估步骤中，更新 critic 以捕捉 soft Q 函数；在策略改进步骤中，根据更新后的 soft Q 函数调整 actor。本文提出一种使用基于能量的归一化流（Energy-Based Normalizing Flows，EBFlow）建模的新型 MaxEnt RL 框架。该框架将策略评估步骤与策略改进步骤整合起来，形成单目标训练过程。我们的方法能够计算策略评估目标中使用的 soft 价值函数，而无需 Monte Carlo 近似。此外，该设计支持对多模态动作分布进行建模，同时便于高效的动作采样。为评估方法性能，我们在 MuJoCo 基准套件以及由 Omniverse Isaac Gym 模拟的多项高维机器人任务上进行了实验。评估结果表明，与广泛采用的代表性基线相比，我们的方法取得了更优的性能。
英文摘要：
Existing Maximum-Entropy (MaxEnt) Reinforcement Learning (RL) methods for continuous action spaces are typically formulated based on actor-critic frameworks and optimized through alternating steps of policy evaluation and policy improvement. In the policy evaluation steps, the critic is updated to capture the soft Q-function. In the policy improvement steps, the actor is adjusted in accordance with the updated soft Q-function. In this paper, we introduce a new MaxEnt RL framework modeled using Energy-Based Normalizing Flows (EBFlow). This framework integrates the policy evaluation steps and the policy improvement steps, resulting in a single objective training process. Our method enables the calculation of the soft value function used in the policy evaluation target without Monte Carlo approximation. Moreover, this design supports the modeling of multi-modal action distributions while facilitating efficient action sampling. To evaluate the performance of our method, we conducted experiments on the MuJoCo benchmark suite and a number of high-dimensional robotic tasks simulated by Omniverse Isaac Gym. The evaluation results demonstrate that our method achieves superior performance compared to widely-adopted representative baselines.
PolicyFlow: Policy Optimization with Continuous Normalizing Flow in Reinforcement Learning
Hong Kong University of Science and Technology（香港科技大学）；Southern University of Science and Technology（南方科技大学）；Zhejiang University-University of Illinois Urbana-Champaign Institute（浙江大学-伊利诺伊大学厄巴纳-香槟分校联合研究所）；LimX Dynamics
A Continuous-Time Actor-Critic Framework for Jump-Diffusion Controls with Normalizing Flows
Yau Mathematical Sciences Center, Tsinghua University（清华大学丘成桐数学科学中心）；Department of Mathematics, Tsinghua University（清华大学数学系）；Department of Mathematics, University of California, Santa Barbara（加州大学圣塔芭芭拉分校数学系）；Department of Statistics and Applied Probability, University of California, Santa Barbara（加州大学圣塔芭芭拉分校统计与应用概率系）；Yanqi Lake Beijing Institute of Mathematical Sciences and Applications（北京雁栖湖应用数学研究院）
Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policies
McGill University（麦吉尔大学）；Mila（米拉研究所）
已知用于连续动作空间的深度强化学习（DRL）算法对于超参数是脆弱的，并且\cut{being}样本效率低下。Soft Actor Critic（SAC）在最大熵RL框架内提出了一种离线深度演员评论算法，该算法提供了更大的稳定性和经验增益。策略分布的选择，一个因子高斯，是由\cut{chosen due}它的容易重新参数化，而不是它的建模能力。我们在SAC框架内引入了规范化流策略，该策略比简单的因子高斯算法更具表达性。\cut{We also present a series of stabilization tricks that enable effective training of these policies in the RL setting.}我们在连续网格世界任务上的经验表明，我们的方法增加了稳定性，更适合于稀疏奖励设置中的困难探索。
Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models
Department of Mathematics, UCLA（UCLA数学系）；Department of Mathematical Sciences, Seoul National University（首尔国立大学数学科学系）；KRAFTON；Department of Linguistics, Stanford University（斯坦福大学语言学系）；Department of Computer Science and Engineering, Santa Clara University（圣克拉拉大学计算机科学与工程系）
Viability of Future Actions: Robust Safety in Reinforcement Learning via Entropy Regularization
Institute for Data Science in Mechanical Engineering, RWTH Aachen University, Germany（机械工程数据科学研究所，亚琛RWTH大学，德国）；Learning Systems and Robotics Lab, Technical University of Munich, Germany（学习系统与机器人实验室，慕尼黑技术大学，德国）
Max-Entropy Reinforcement Learning with Flow Matching and A Case Study on LQR
School of Engineering and Applied Sciences, Harvard University（哈佛大学工程与应用科学学院）；Harvard University（哈佛大学）；College of Computing, Georgia Institute of Technology（佐治亚理工学院计算机学院）
Entropy-Regularized Reinforcement Learning for Linear-Quadratic Stackelberg Differential Games in Regime-Switching Diffusion Models
School of Mathematics and Statistics, Anhui Normal University（安徽师范大学数学与统计学学院）；School of Statistics, Key Laboratory of Advanced Theory and Application in Statistics and Data Science-MOE, Faculty of Economics and Management, East China Normal University（东华大学统计学院）；School of Mathematics, Southeast University（东南大学数学学院）
ME-IGM: Individual-Global-Max in Maximum Entropy Multi-Agent Reinforcement Learning
Carnegie Mellon University（卡内基梅隆大学）；Zhejiang University（浙江大学）；XPeng Inc.（XPeng公司）；The University of Hong Kong（香港大学）；INFIFORCE Intelligent Tech. Co., Ltd.（INFIFORCE智能科技有限公司）
Entropy Regularized Reinforcement Learning for Zero-Sum Stochastic Differential Games in a Regime-Switching Jump-Diffusion Process
School of Mathematics and Statistics（数学与统计学学院）；Department of Actuarial Studies and Business Analytics（精算学与商业分析系）；School of Statistics, KLATASDS-MOE, China Inclusive Ageing Finance Research Center（统计学学院，KLATASDS-MOE，中国包容性老龄化金融研究中心）；East China Normal University（华东师范大学）
Tracking Drift: Variation-Aware Entropy Scheduling for Non-Stationary Reinforcement Learning
School of Future Technology, Southeast University, Nanjing, China（东南大学未来技术学院，南京，中国）；School of Automation, Southeast University, Nanjing, China（东南大学自动化学院，南京，中国）
