Discretizing Continuous Action Space with Unimodal Probability Distributions for On-Policy Reinforcement Learning
Nanjing University(南京大学);Research Center for Novel Technology of Intelligent Equipment, Nanjing University(南京大学智能装备新技术研究中心);Chinese Academy of Sciences(中国科学院);University of Chinese Academy of Sciences(中国科学院大学)
For on-policy reinforcement learning, discretizing action space for continuous control can easily express multiple modes and is straightforward to optimize. However, without considering the inherent ordering between the discrete atomic actions, the explosion in the number of discrete actions can possess undesired properties and induce a higher variance for the policy gradient estimator. In this paper, we introduce a straightforward architecture that addresses this issue by constraining the discrete policy to be unimodal using Poisson probability distributions. This unimodal architecture can better leverage the continuity in the underlying continuous action space using explicit unimodal probability distributions. We conduct extensive experiments to show that the discrete policy with the unimodal probability distribution provides significantly faster convergence and higher performance for on-policy reinforcement learning algorithms in challenging control tasks, especially in highly complex tasks such as Humanoid. We provide theoretical analysis on the variance of the policy gradient estimator, which suggests that our attentively designed unimodal discrete policy can retain a lower variance and yield a stable learning process.
Distributional Soft Actor-Critic with Diffusion Policy
School of Vehicle and Mobility, Tsinghua University(车辆与移动学院,清华大学);School of Mechanical Engineering, University of Science and Technology Beijing(机械工程学院,北京科技大学)
Maximum Entropy Reinforcement Learning with Diffusion Policy
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);AiRiA
Omni-Perception Policy Optimization for Multimodal Emotion Reasoning
University of Science and Technology of China(中国科学技术大学);SenseTime Research(商汤科技研究院);National University of Singapore(新加坡国立大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院)
Dual-Flow Reinforcement Learning with State-Aware Exploration
State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University(智能绿色车辆与移动国家重点实验室,清华大学);Tsinghua University-Toyota Joint Center(清华大学-丰田联合中心);Tsinghua University–SAIC GM Wuling Joint Research Center(清华大学-上汽通用五菱联合研究中心)
Efficient Anti-exploration via VQVAE and Fuzzy Clustering in Offline Reinforcement Learning
State Key Laboratory of Multimodal Artificial Intelligence Systems(多模态人工智能系统国家重点实验室);State Key Laboratory of Management and Control for Complex Systems(复杂系统管理与控制国家重点实验室);Chinese Academy of Sciences(中国科学院);WAYTOUS Inc.(WAYTOUS公司);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东人工智能与数字经济实验室(深圳));Institute of Artificial Intelligence and Robotics(人工智能与机器人研究院);Xi’an Jiaotong University(西安交通大学);School of Artificial Intelligence(人工智能学院);Key Laboratory of Intelligent Sensing System and Security (Hubei University)(智能感知系统与安全重点实验室(湖北大学));Ministry of Education(教育部);Department of Electrical and Computer Engineering(电气与计算机工程系)
Nanyang Technological University(南洋理工大学);National University of Singapore(国立新加坡大学);Carnegie Mellon University(卡内基梅隆大学);CFAR Agency for Science Technology and Research(科技研究局(CFAR));IHPC Agency for Science Technology and Research(科技研究局(IHPC))
How Does the Lagrangian Guide Safe Reinforcement Learning through Diffusion Models?
University College London(伦敦大学学院);Imperial College London(伦敦帝国学院);University of California, San Diego(加州大学圣地亚哥分校);Kyoto University(京都大学);King's College London(伦敦国王学院);University of Sheffield(谢菲尔德大学);Shanghai University of Finance and Economics(上海财经大学)