Modern Deep Reinforcement Learning (RL) algorithms require estimates of the maximal Q-value, which are difficult to compute in continuous domains with an infinite number of possible actions. In this work, we introduce a new update rule for online and offline RL which directly models the maximal value using Extreme Value Theory (EVT), drawing inspiration from economics. By doing so, we avoid computing Q-values using out-of-distribution actions which is often a substantial source of error. Our key insight is to introduce an objective that directly estimates the optimal soft-value functions (LogSumExp) in the maximum entropy RL setting without needing to sample from a policy. Using EVT, we derive our \emph{Extreme Q-Learning} framework and consequently online and, for the first time, offline MaxEnt Q-learning algorithms, that do not explicitly require access to a policy or its entropy. Our method obtains consistently strong performance in the D4RL benchmark, outperforming prior works by \emph{10+ points} on the challenging Franka Kitchen tasks while offering moderate improvements over SAC and TD3 on online DM Control tasks. Visualizations and code can be found on our website at https://div99.github.io/XQL/.
Maximum Entropy Reinforcement Learning with Diffusion Policy
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);AiRiA
Boosting Maximum Entropy Reinforcement Learning via One-Step Flow Matching
Tianjin Key Laboratory of Intelligent Unmanned Swarm Technology(天津智能无人群技术与系统重点实验室);School of Electrical and Information Engineering(电气与信息工程学院);Tianjin University(天津大学)
Maximum Entropy Heterogeneous-Agent Reinforcement Learning
Institute for AI, Peking University(北京大学人工智能研究院);National Key Laboratory of General AI, BIGAI(北京智源人工智能研究院通用人工智能全国重点实验室);University of Technology Sydney(悉尼科技大学);Tencent AI Lab(腾讯人工智能实验室)
S$^2$AC: Energy-Based Reinforcement Learning with Stein Soft Actor Critic
Qatar Computing Research Institute, Hamad Bin Khalifa University(卡塔尔计算研究所、哈马德·本·哈利法大学);School of Computer Science and Engineering, Nanyang Technological University(南洋理工大学计算机科学与工程学院);SalesForce(SalesForce(赛富时));Skywork AI;Data Science, William & Mary(威廉玛丽学院数据科学系)
Efficient first-order algorithms for large-scale, non-smooth maximum entropy models with application to wildfire science
Courant Institute of Mathematical Sciences, New York University(纽约大学柯朗数学科学研究所);Department of Earth and Environmental Engineering, Columbia University(哥伦比亚大学地球与环境工程系);Division of Applied Mathematics, Brown University(布朗大学应用数学系)
Intelligent Sampling of Extreme-Scale Turbulence Datasets for Accurate and Efficient Spatiotemporal Model Training
Oak Ridge National Laboratory(橡树岭国家实验室);Argonne National Laboratory(阿贡国家实验室);York University(约克大学);University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校);Georgia Institute of Technology(佐治亚理工学院)
GEM-T: Generative Tabular Data via Fitting Moments
Department of Pathology at Beth Israel Deaconess Medical Center (BIDMC)(波士顿贝塞斯达德aconess医疗中心病理学系);Department of Medicine, the Bakar Computational Health Sciences Institute, and the UCSF UC Berkeley Joint Program for Computational Precision Health at the University of California San Francisco(旧金山大学医学系、Bakar计算健康科学研究所及UCSF-伯克利联合计算精准健康项目);Department of Pathology and the Division of Clinical Informatics, Department of Medicine, BIDMC and with Harvard Medical School(波士顿贝塞斯达德aconess医疗中心病理学系及医学系临床信息学分会、哈佛医学院)
GEM-T 基于最大熵原理直接拟合表格数据列间 n 阶交互,以极少的可训练参数在 34 个数据集中 68% 上达到或超越深度神经网络方法,为结构化数据提供轻量高性能生成新方向。