The Mean Square Error (MSE) is commonly utilized to estimate the solution of the optimal value function in the vast majority of offline reinforcement learning (RL) models and has achieved outstanding performance. However, we find that its principle can lead to overestimation phenomenon for the value function. In this paper, we first theoretically analyze overestimation phenomenon led by MSE and provide the theoretical upper bound of the overestimated error. Furthermore, to address it, we propose a novel Bellman underestimated operator to counteract overestimation phenomenon and then prove its contraction characteristics. At last, we propose the offline RL algorithm based on underestimated operator and diffusion policy model. Extensive experimental results on D4RL tasks show that our method can outperform state-of-the-art offline RL algorithms, which demonstrates that our theoretical analysis and underestimation way are effective for offline RL tasks.
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Salesforce Research（Salesforce研究院）
Recent theoretical work studies sample-efficient reinforcement learning (RL) extensively in two settings: learning interactively in the environment (online RL), or learning from an offline dataset (offline RL). However, existing algorithms and theories for learning near-optimal policies in these two settings are rather different and disconnected. Towards bridging this gap, this paper initiates the theoretical study of policy finetuning, that is, online RL where the learner has additional access to a "reference policy" $μ$ close to the optimal policy $π_\star$ in a certain sense. We consider th…
Reinforcement learning (RL) has been shown to be effective at learning control from experience. However, RL typically requires a large amount of online interaction with the environment. This limits its applicability to real-world settings, such as in robotics, where such interaction is expensive. In this work we investigate ways to minimize online interactions in a target task, by reusing a suboptimal policy we might have access to, for example from training on related prior tasks, or in simulation. To this end, we develop two RL algorithms that can speed up training by using not only the acti…
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Salesforce AI Research（Salesforce AI研究院）；The Hong Kong University of Science and Technology（香港科技大学）
Offline reinforcement learning (Offline RL) suffers from the innate distributional shift as it cannot interact with the physical environment during training. To alleviate such limitation, state-based offline RL leverages a learned dynamics model from the logged experience and augments the predicted state transition to extend the data distribution. For exploiting such benefit also on the image-based RL, we firstly propose a generative model, S2P (State2Pixel), which synthesizes the raw pixel of the agent from its corresponding state. It enables bridging the gap between the state and the image d…
Knowledge Representation is important issue in reinforcement learning. In this paper, we bridge the gap between reinforcement learning and knowledge representation, by providing a rich knowledge representation framework, based on normal logic programs with answer set semantics, that is capable of solving model-free reinforcement learning problems for more complex do-mains and exploits the domain-specific knowledge. We prove the correctness of our approach. We show that the complexity of finding an offline and online policy for a model-free reinforcement learning problem in our approach is NP-c…
Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization
Institute for AI Industry Research (AIR), Tsinghua University（清华大学智能产业研究院（AIR））；Beijing Jiaotong University（北京交通大学）；UT Austin（德克萨斯大学奥斯汀分校）；Tsinghua University（清华大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Offline reinforcement learning, which aims at optimizing sequential decision-making strategies with historical data, has been extensively applied in real-life applications. State-Of-The-Art algorithms usually leverage powerful function approximators (e.g. neural networks) to alleviate the sample complexity hurdle for better empirical performances. Despite the successes, a more systematic understanding of the statistical complexity for function approximation remains lacking. Towards bridging the gap, we take a step by considering offline reinforcement learning with differentiable function class…
Polytechnique Montréal（蒙特利尔理工学院）；IRT Saint Exupéry Canada（加拿大圣埃克苏佩里技术研究所）；Ubisoft La Forge（育碧La Forge实验室）；Inria Bordeaux Sud-Ouest（法国国家信息与自动化研究所波尔多西南分部）
Learning efficiently a causal model of the environment is a key challenge of model-based RL agents operating in POMDPs. We consider here a scenario where the learning agent has the ability to collect online experiences through direct interactions with the environment (interventional data), but has also access to a large collection of offline experiences, obtained by observing another agent interacting with the environment (observational data). A key ingredient, that makes this situation non-trivial, is that we allow the observed agent to interact with the environment based on hidden informatio…
Optimal Control of Probabilistic Dynamics Models via Mean Hamiltonian Minimization
Department of Machine Learning and Neural Computing, Donders Institute for Brain, Cognition and Behaviour, Radboud University（机器学习与神经计算系，多纳尔斯脑认知行为研究所，拉德堡德大学）；Cluster of Excellence Machine Learning, University of Tübingen（卓越机器学习集群，图宾根大学）；Department of Engineering, Control Group, University of Oxford（工程系，控制组，牛津大学）
We consider the problem of off-policy evaluation for reinforcement learning, where the goal is to estimate the expected reward of a target policy $π$ using offline data collected by running a logging policy $μ$. Standard importance-sampling based approaches for this problem suffer from a variance that scales exponentially with time horizon $H$, which motivates a splurge of recent interest in alternatives that break the "Curse of Horizon" (Liu et al. 2018, Xie et al. 2019). In particular, it was shown that a marginalized importance sampling (MIS) approach can be used to achieve an estimation er…
Data-driven simulator of multi-animal behavior with unknown dynamics via offline and online reinforcement learning
Graduate School of Informatics, Nagoya University, Japan（名古屋大学信息学研究科）；RIKEN Center for Advanced Intelligence Project, Japan（RIKEN高级智能项目研究中心）；Graduate School of Arts and Sciences, The University of Tokyo, Japan（东京大学文学系研究科）；Project team for SIP, Japan（SIP项目团队）；Japan Agency for Marine-Earth Science and Technology, Japan（日本海洋地球科学技术机构）；Faculty of Education, Shitennoji University, Japan（世田谷大学教育学部）；Graduate School of Engineering, The University of Tokyo, Japan（东京大学工学研究科）；Graduate School of Science and Technology, Niigata University, Japan（新潟大学科学技术研究科）；Graduate School of Science, Nagoya University, Japan（名古屋大学理学研究科）；Principles of Informatics Research Division, National Institute of Informatics, Japan（信息学原理研究部门，日本信息处理技术研究所）；Graduate School of Information Science, The University of Osaka, Japan（大阪大学信息科学研究科）
On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
Southeast University（东南大学）；University of California, Los Angeles（加州大学洛杉矶分校）；Shanghai Jiao Tong University（上海交通大学）；Nanyang Technological University（南洋理工大学）；University of California, Berkeley（加州大学伯克利分校）；Wuhan University（武汉大学）；University of California, Merced（加州大学默塞德分校）
Offline Model-Based Optimization by Learning to Rank
National Key Laboratory for Novel Software Technology（新型软件技术国家重点实验室）；School of Artificial Intelligence（人工智能学院）；Key Laboratory of Water Big Data Technology of Ministry of Water Resources（水利部水大数据技术重点实验室）；College of Computer Science and Software Engineering（计算机科学与软件工程学院）；Advanced Computing and Storage Lab（先进计算与存储实验室）；Huawei Technologies Co., Ltd.（华为技术有限公司）
RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignment
Shenzhen Research Institute of Big Data（深圳大数据研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Qwen LLM Application Team, Alibaba（阿里巴巴Qwen大模型应用团队）；Stanford University（斯坦福大学）
Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization
School of Computer Science and Engineering, Southeast University, Nanjing, China（东南大学计算机科学与工程学院）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（新一代人工智能技术及其交叉应用国家重点实验室）
Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directions
Department of Computer Science, Harvard University（哈佛大学计算机科学系）；Department of Statistics, University of Wisconsin–Madison（威斯康星大学麦迪逊分校统计学系）；Department of Statistics, Harvard University（哈佛大学统计学系）；School of Data Science and Society, University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校数据科学与社会学院）；Department of Mathematics, Imperial College London（伦敦帝国理工学院数学系）
Navigating the Noise: Bringing Clarity to ML Parameterization Design with O(100) Ensembles
Expertise Center for Climate Extremes, University of Lausanne（洛桑大学气候极端事件专业中心）；NVIDIA Research（英伟达研究院）；Department of Biomedical Data Science, Stanford University School of Medicine（斯坦福大学医学院生物医学数据科学系）；Department of Earth System Sciences, University of California at Irvine（加州大学欧文分校地球系统科学系）；Multimodal Cognitive AI, Intel Labs（英特尔实验室多模态认知人工智能部门）；Faculty of Geosciences and Environment, University of Lausanne（洛桑大学地球科学与环境学院）；Department of Statistics, University of California at Irvine（加州大学欧文分校统计系）；Department of Earth and Planetary Sciences, Harvard University（哈佛大学地球与行星科学系）；LEAP Science and Technology Center, School of Engineering and Applied Sciences, Climate School, Columbia University（哥伦比亚大学气候学院工程与应用科学学院LEAP科技中心）；Berkeley AI Research (BAIR), Department of Electrical Engineering and Computer Science, University of California at Berkeley（加州大学伯克利分校电气工程与计算机科学系伯克利人工智能研究中心）
