We consider constrained policy optimization in Reinforcement Learning, where the constraints are in form of marginals on state visitations and global action executions. Given these distributions, we formulate policy optimization as unbalanced optimal transport over the space of occupancy measures. We propose a general purpose RL objective based on Bregman divergence and optimize it using Dykstra's algorithm. The approach admits an actor-critic algorithm for when the state or action space is large, and only samples from the marginals are available. We discuss applications of our approach and provide demonstrations to show the effectiveness of our algorithm.
Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO), as the widely employed policy based reinforcement learning (RL) methods, are prone to converge to a sub-optimal solution as they limit the policy representation to a particular parametric distribution class. To address this issue, we develop an innovative Optimistic Distributionally Robust Policy Optimization (ODRPO) algorithm, which effectively utilizes Optimistic Distributionally Robust Optimization (DRO) approach to solve the trust region constrained optimization problem without parameterizing the policies. Our …
Our goal is to train control policies that generalize well to unseen environments. Inspired by the Distributionally Robust Optimization (DRO) framework, we propose DRAGEN - Distributionally Robust policy learning via Adversarial Generation of ENvironments - for iteratively improving robustness of policies to realistic distribution shifts by generating adversarial environments. The key idea is to learn a generative model for environments whose latent variables capture cost-predictive and realistic variations in environments. We perform DRO with respect to a Wasserstein ball around the empirical…
Affiliated High School of South China Normal University（华南师范大学附属中学）
We consider the problem of offline reinforcement learning with model-based control, whose goal is to learn a dynamics model from the experience replay and obtain a pessimism-oriented agent under the learned model. Current model-based constraint includes explicit uncertainty penalty and implicit conservative regularization that pushes Q-values of out-of-distribution state-action pairs down and the in-distribution up. While the uncertainty estimation, on which the former relies on, can be loosely calibrated for complex dynamics, the latter performs slightly better. To extend the basic idea of re…
Real-world applications require RL algorithms to act safely. During learning process, it is likely that the agent executes sub-optimal actions that may lead to unsafe/poor states of the system. Exploration is particularly brittle in high-dimensional state/action space due to increased number of low-performing actions. In this work, we consider risk-averse exploration in approximate RL setting. To ensure safety during learning, we propose the distributionally robust policy iteration scheme that provides lower bound guarantee on state-values. Our approach induces a dynamic level of risk to preve…
Carnegie Mellon University（卡内基梅隆大学）；Columbia University（哥伦比亚大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
One key challenge for multi-task Reinforcement learning (RL) in practice is the absence of task indicators. Robust RL has been applied to deal with task ambiguity, but may result in over-conservative policies. To balance the worst-case (robustness) and average performance, we propose Group Distributionally Robust Markov Decision Process (GDR-MDP), a flexible hierarchical MDP formulation that encodes task groups via a latent mixture model. GDR-MDP identifies the optimal policy that maximizes the expected return under the worst-possible qualified belief over task groups within an ambiguity set. …
We discuss the thought-provoking new objective functions for policy learning that were proposed in "More efficient policy learning via optimal retargeting" by Nathan Kallus and "Learning optimal distributionally robust individualized treatment rules" by Weibin Mo, Zhengling Qi, and Yufeng Liu. We show that it is important to take the curvature of the value function into account when working within the retargeting framework, and we introduce two ways to do so. We also describe more efficient approaches for leveraging calibration data when learning distributionally robust policies.
Distributionally Robust Federated Learning with Outlier Resilience
Division of Decision and Control Systems, School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology（瑞典皇家理工学院电气工程与计算机科学系决策与控制系统 division）；Department of Control Science and Engineering, College of Electronics and Information Engineering, Tongji University（同济大学电子信息工程学院控制科学与工程系）；Mechanical Engineering and Material Science, Duke University（达特茅斯大学机械工程与材料科学系）
Stanford University（斯坦福大学）；University of California, Los Angeles（加州大学洛杉矶分校）；New York University（纽约大学）
With the rise of the digital economy and an explosion of available information about consumers, effective personalization of goods and services has become a core business focus for companies to improve revenues and maintain a competitive edge. This paper studies the personalization problem through the lens of policy learning, where the goal is to learn a decision-making rule (a policy) that maps from consumer and product characteristics (features) to recommendations (actions) in order to optimize outcomes (rewards). We focus on using available historical data for offline learning with unknown …
Tsinghua University（清华大学）；Hong Kong University of Science and Technology（香港科技大学）；Stanford University（斯坦福大学）；Sun Yat-sen University（中山大学）；New York University（纽约大学）
Among the reasons hindering reinforcement learning (RL) applications to real-world problems, two factors are critical: limited data and the mismatch between the testing environment (real environment in which the policy is deployed) and the training environment (e.g., a simulator). This paper attempts to address these issues simultaneously with distributionally robust offline RL, where we learn a distributionally robust policy using historical data obtained from the source environment by optimizing against a worst-case perturbation thereof. In particular, we move beyond tabular settings and con…
Wasserstein Distributionally Robust Regret Optimization for Reinforcement Learning from Human Feedback
Department of Statistics and Operations Research, University of North Carolina（统计与运筹学系，北卡罗来纳大学）；Imperial Business School, Imperial College London（帝国理工学院伦敦商学院）；Department of Management Science and Engineering, Stanford University（管理科学与工程系，斯坦福大学）
Cornell University（康奈尔大学）；Cornell Tech（康奈尔科技校区）；Tsinghua University（清华大学）；Arena Technologies（Arena科技公司）；New York University（纽约大学）
Off-policy evaluation and learning (OPE/L) use offline observational data to make better decisions, which is crucial in applications where online experimentation is limited. However, depending entirely on logged data, OPE/L is sensitive to environment distribution shifts -- discrepancies between the data-generating environment and that where policies are deployed. \citet{si2020distributional} proposed distributionally robust OPE/L (DROPE/L) to address this, but the proposal relies on inverse-propensity weighting, whose estimation error and regret will deteriorate if propensities are nonparamet…
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；George Washington University（乔治华盛顿大学）
We thank the opportunity offered by editors for this discussion and the discussants for their insightful comments and thoughtful contributions. We also want to congratulate Kallus (2020) for his inspiring work in improving the efficiency of policy learning by retargeting. Motivated from the discussion in Dukes and Vansteelandt (2020), we first point out interesting connections and distinctions between our work and Kallus (2020) in Section 1. In particular, the assumptions and sources of variation for consideration in these two papers lead to different research problems with different scopes an…
Distributionally Robust Imitation Learning: Layered Control Architecture for Certifiable Autonomy
Department of Mechanical Science and Engineering, University of Illinois Urbana-Champaign（机械科学与工程系，伊利诺伊大学厄巴纳-香槟分校）；Department of Electrical and Systems Engineering, University of Pennsylvania（电气与系统工程系，宾夕法尼亚大学）；Mechanical and Civil Engineering, Control and Dynamical Systems, California Institute of Technology（机械与土木工程、控制与动态系统，加州理工学院）；Massachusetts Institute of Technology（麻省理工学院）；Advanced Technology Labs, Lockheed Martin（洛克希德·马丁高级技术实验室）
The rise of big data analytics has automated the decision-making of companies and increased supply chain agility. In this paper, we study the supply chain contract design problem faced by a data-driven supplier who needs to respond to the inventory decisions of the downstream retailer. Both the supplier and the retailer are uncertain about the market demand and need to learn about it sequentially. The goal for the supplier is to develop data-driven pricing policies with sublinear regret bounds under a wide range of possible retailer inventory policies for a fixed time horizon. To capture the d…
Learning Optimal Distributionally Robust Individualized Treatment Rules Integrating Multi-Source Data
Department of Applied Mathematics, The Hong Kong Polytechnic University（应用数学系，香港理工大学）；Department of Biostatistics, City University of Hong Kong（生物统计学系，香港城市大学）
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；George Washington University（乔治华盛顿大学）
Recent development in the data-driven decision science has seen great advances in individualized decision making. Given data with individual covariates, treatment assignments and outcomes, policy makers best individualized treatment rule (ITR) that maximizes the expected outcome, known as the value function. Many existing methods assume that the training and testing distributions are the same. However, the estimated optimal ITR may have poor generalizability when the training and testing distributions are not identical. In this paper, we consider the problem of finding an optimal ITR from a re…
Hong Kong University of Science and Technology（香港科技大学）；Tsinghua University（清华大学）；Stanford University（斯坦福大学）；New York University（纽约大学）；Arena Technologies（Arena Technologies公司）
Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithms
Department of Management Science and Engineering, Stanford University（斯坦福大学管理科学与工程系）；Center for Data Science, Peking University（北京大学数据科学中心）；Department of Computer Science, University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校计算机科学系）
Learning Optimal Distributionally Robust Stochastic Control in Continuous State Spaces
Department of Industrial and Systems Engineering（工业与系统工程系）；USC（南加州大学）；Management Science and Engineering（管理科学与工程）；Stanford University（斯坦福大学）；Department of Industrial Engineering and Decision Analytics（工业工程与决策分析系）；HKUST（香港科技大学）；Stern School of Business（Stern 商学院）
Stony Brook University（石溪大学）；Institute of Semiconductors, Chinese Academy of Sciences（中国科学院半导体研究所）
We consider the problem of costly feature classification, where we sequentially select the subset of features to make a balance between the classification error and the feature cost. In this paper, we first cast the task into a MDP problem and use Advantage Actor Critic algorithm to solve it. In order to further improve the agent's performance and make the policy explainable, we employ the Monte Carlo Tree Search to update the policy iteratively. During the procedure, we also consider its performance on the unbalanced dataset and its sensitivity to the missing value. We evaluate our model on m…
The recent emergence of reinforcement learning has created a demand for robust statistical inference methods for the parameter estimates computed using these algorithms. Existing methods for statistical inference in online learning are restricted to settings involving independently sampled observations, while existing statistical inference methods in reinforcement learning (RL) are limited to the batch setting. The online bootstrap is a flexible and efficient approach for statistical inference in linear stochastic approximation algorithms, but its efficacy in settings involving Markov noise, s…
The University of Alabama（阿拉巴马大学）；Purdue University（普渡大学）
In a ride-hailing system, an optimal relocation of vacant vehicles can significantly reduce fleet idling time and balance the supply-demand distribution, enhancing system efficiency and promoting driver satisfaction and retention. Model-free deep reinforcement learning (DRL) has been shown to dynamically learn the relocating policy by actively interacting with the intrinsic dynamics in large-scale ride-hailing systems. However, the issues of sparse reward signals and unbalanced demand and supply distribution place critical barriers in developing effective DRL models. Conventional exploration s…
