Robots and animals both experience the world through their bodies and senses. Their embodiment constrains their experiences, ensuring they unfold continuously in space and time. As a result, the experiences of embodied agents are intrinsically correlated. Correlations create fundamental challenges for machine learning, as most techniques rely on the assumption that data are independent and identically distributed. In reinforcement learning, where data are directly collected from an agent's sequential experiences, violations of this assumption are often unavoidable. Here, we derive a method that overcomes this issue by exploiting the statistical mechanics of ergodic processes, which we term maximum diffusion reinforcement learning. By decorrelating agent experiences, our approach provably enables single-shot learning in continuous deployments over the course of individual task attempts. Moreover, we prove our approach generalizes well-known maximum entropy techniques, and robustly exceeds state-of-the-art performance across popular benchmarks. Our results at the nexus of physics, learning, and control form a foundation for transparent and reliable decision-making in embodied reinforcement learning agents.
Offline Reinforcement Learning with Discrete Diffusion Skills
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Individual Control Barrier Functions-Guided Diffusion Model for Safe Offline Multi-Agent Reinforcement Learning
Department of Electrical Engineering and Automation, Aalto University(阿尔托大学电气工程与自动化系);School of Computing and Data Science, Xiamen University Malaysia(厦门大学马来西亚分校计算与数据科学学院);Department of Computer Science, University of Toronto(多伦多大学计算机科学系)
Connections between reinforcement learning with feedback,test-time scaling, and diffusion guidance: An anthology
Department of Statistics and Data Science, Chinese University of Hong Kong(统计与数据科学系,香港中文大学);Department of Statistics and Data Science, the Wharton School, University of Pennsylvania(统计与数据科学系,沃顿商学院,宾夕法尼亚大学)
TeViR: Text-to-Video Reward with Diffusion Models for Efficient Reinforcement Learning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Maximum Entropy Reinforcement Learning with Diffusion Policy
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);AiRiA
Dependency-Aware CAV Task Scheduling via Diffusion-Based Reinforcement Learning
Frontier Research Center, Pengcheng Laboratory(鹏城实验室前沿研究中心);School of Information and Communication Engineering, Beijing University of Posts and Telecommunications(北京邮电大学信息与通信工程学院)
LacaDM: A Latent Causal Diffusion Model for Multiobjective Reinforcement Learning
School of Information Science and Technology, Guangdong University of Foreign Studies(广东外语大学信息科学与技术学院);School of Engineering, Westlake University(西湖大学工程学院)
Generalized Contrastive Divergence: Joint Training of Energy-Based Model and Diffusion Model through Inverse Reinforcement Learning
Korea Institute for Advanced Study(韩国高等研究院);University of Seoul(首尔市立大学);Seoul National University(首尔国立大学);Hanyang University(汉阳大学);Saige Research(赛奇研究院)
Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learning
Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University Munich(慕尼黑机器人与机器智能研究所(MIRMI),技术大学慕尼黑);Practical Project Student Exchange Program, Technical University Munich(技术大学慕尼黑实践项目学生交换计划);MIT World Peace University(MIT和平大学)
Guiding Diffusion Models with Reinforcement Learning for Stable Molecule Generation
Fudan University(复旦大学);Shanghai Innovation Institute(上海创新研究院);Shanghai Academy of Artificial Intelligence for Science(上海人工智能科学研究院);INFTECH;Zhongshan Hospital(中山医院)
DiDrive: A Risk-Aware Hierarchical Diffusion Framework for Safe Offline Reinforcement Learning in Autonomous Driving
College of Computer and Data Science, Fuzhou University(福州大学计算机与数据科学学院);Maynooth International College of Engineering, Fuzhou University(福州大学梅努斯国际工程学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
DAWM: Diffusion Action World Models for Offline Reinforcement Learning via Action-Inferred Transitions
Department of Computer Science, University of Munich, Munich, Germany(慕尼黑大学计算机科学系);Munich Center for Machine Learning (MCML), Munich, Germany(慕尼黑机器学习中心(MCML))