In contrast to classical reinforcement learning (RL), distributional RL algorithms aim to learn the distribution of returns rather than their expected value. Since the nature of the return distribution is generally unknown a priori or arbitrarily complex, a common approach finds approximations within a set of representable, parametric distributions. Typically, this involves a projection of the unconstrained distribution onto the set of simplified distributions. We argue that this projection step entails a strong inductive bias when coupled with neural networks and gradient descent, thereby profoundly impacting the generalization behavior of learned models. In order to facilitate reliable uncertainty estimation through diversity, we study the combination of several different projections and representations in a distributional ensemble. We establish theoretical properties of such projection ensembles and derive an algorithm that uses ensemble disagreement, measured by the average 1-Wasserstein distance, as a bonus for deep exploration. We evaluate our algorithm on the behavior suite benchmark and VizDoom and find that diverse projection ensembles lead to significant performance improvements over existing methods on a variety of tasks with the most pronounced gains in directed exploration problems.
Department of Mathematics, Linn us University, 351 95 Växjö, Sweden
It is well known that ensemble methods often provide enhanced performance in reinforcement learning. In this paper, we explore this concept further by using group-aided training within the distributional reinforcement learning paradigm. Specifically, we propose an extension to categorical reinforcement learning, where distributional learning targets are implicitly based on the total information gathered by an ensemble. We empirically show that this may lead to much more robust initial learning, a stronger individual performance level, and good efficiency on a per-sample basis.
Zenseact AB（Zenseact公司）；Chalmers University of Technology（查尔姆斯理工大学）；Scool INRIA Lille-Nord Europe（法国国家信息与自动化研究所里尔-北欧洲Scool团队）；CRIStAL, CNRS（法国国家科学研究中心CRIStAL实验室）；University of Neuchatel（纳沙泰尔大学）；University of Oslo（奥斯陆大学）
In this paper, we consider risk-sensitive sequential decision-making in Reinforcement Learning (RL). Our contributions are two-fold. First, we introduce a novel and coherent quantification of risk, namely composite risk, which quantifies the joint effect of aleatory and epistemic risk during the learning process. Existing works considered either aleatory or epistemic risk individually, or as an additive combination. We prove that the additive formulation is a particular case of the composite risk when the epistemic risk measure is replaced with expectation. Thus, the composite risk is more sen…
New York University（纽约大学）；New York University Shanghai（上海纽约大学）；University of California, Berkeley（加州大学伯克利分校）
Recent advances in model-free deep reinforcement learning (DRL) show that simple model-free methods can be highly effective in challenging high-dimensional continuous control tasks. In particular, Truncated Quantile Critics (TQC) achieves state-of-the-art asymptotic training performance on the MuJoCo benchmark with a distributional representation of critics; and Randomized Ensemble Double Q-Learning (REDQ) achieves high sample efficiency that is competitive with state-of-the-art model-based methods using a high update-to-data ratio and target randomization. In this paper, we propose a novel mo…
Foundations of Multivariate Distributional Reinforcement Learning
Mila — Québec AI Institute（米拉-魁北克人工智能研究所）；McGill University（麦吉尔大学）；Google DeepMind（谷歌DeepMind）；Gatsby Unit, University College London（伦敦大学学院盖茨比计算神经科学单元）
Quantile-Coupled Flow Matching for Distributional Reinforcement Learning
Oxford Robotics Institute, University of Oxford（牛津大学机器人研究所）；Bristol Robotics Laboratory, University of the West of England（布里斯托尔机器人实验室，西英格兰大学）；Dyson Institute of Engineering & Technology（戴森工程与技术研究院）
Distributional reinforcement learning (DRL) extends the value-based approach by approximating the full distribution over future returns instead of the mean only, providing a richer signal that leads to improved performances. Quantile Regression (QR) based methods like QR-DQN project arbitrary distributions into a parametric subset of staircase distributions by minimizing the 1-Wasserstein distance. However, due to biases in the gradients, the quantile regression loss is used instead for training, guaranteeing the same minimizer and enjoying unbiased gradients. Non-crossing constraints on the q…
Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models
Qualcomm AI Research, Qualcomm Vietnam Company Limited（高通人工智能研究, 高通越南公司）；National University of Singapore, Singapore（新加坡国立大学）；Monash University, Australia（墨尔本大学）
Distributional Regression U-Nets for the Postprocessing of Precipitation Ensemble Forecasts
Université de Franche Comté（弗朗什-孔泰大学）；CNRS（法国国家科学研究中心）；Laboratoire des Sciences du Climat et de l’Environnement（气候与环境科学实验室）；CEA（法国原子能和替代能源委员会）；UVSQ（凡尔赛大学）；IPSL（巴黎地球物理研究所）；CNRM（法国国家气象研究中心）；Université de Toulouse（图卢兹大学）；Météo-France（法国气象局）
The Chinese University of Hong Kong（香港中文大学）；Nanjing University（南京大学）
Wasserstein \textbf{D}istributionally \textbf{R}obust \textbf{O}ptimization (DRO) is concerned with finding decisions that perform well on data that are drawn from the worst-case probability distribution within a Wasserstein ball centered at a certain nominal distribution. In recent years, it has been shown that various DRO formulations of learning models admit tractable convex reformulations. However, most existing works propose to solve these convex reformulations by general-purpose solvers, which are not well-suited for tackling large-scale problems. In this paper, we focus on a family of W…
VinAI Research（VinAI研究院）；University of Texas, Austin（得克萨斯大学奥斯汀分校）
Sliced-Wasserstein distance (SW) and its variant, Max Sliced-Wasserstein distance (Max-SW), have been used widely in the recent years due to their fast computation and scalability even when the probability measures lie in a very high dimensional space. However, SW requires many unnecessary projection samples to approximate its value while Max-SW only uses the most important projection, which ignores the information of other useful directions. In order to account for these weaknesses, we propose a novel distance, named Distributional Sliced-Wasserstein distance (DSW), that finds an optimal dist…
Abdul Wali Khan University（阿卜杜勒·瓦利·汗大学）；King Abdul Aziz University（阿卜杜勒·阿齐兹国王大学）
Ensembles based on k nearest neighbours (kNN) combine a large number of base learners, each constructed on a sample taken from a given training data. Typical kNN based ensembles determine the k closest observations in the training data bounded to a test sample point by a spherical region to predict its class. In this paper, a novel random projection extended neighbourhood rule (RPExNRule) ensemble is proposed where bootstrap samples from the given training data are randomly projected into lower dimensions for additional randomness in the base models and to preserve features information. It use…
Beyond Discriminant Patterns: On the Robustness of Decision Rule Ensembles
School of Computing；Information Technology Great Bay University Dongguan, China；School of Informatics University of Edinburgh Edinburgh, UK；DAI Cluster Eindhoven University of Technology Eindhoven, Netherlands；Department of Computer Science Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, UAE；Department of Mathematics；Computer Science Eindhoven University of Technology Eindhoven, Netherlands
University of Massachusetts Dartmouth（马萨诸塞大学达特茅斯分校）；Dali University（大理大学）
K-nearest neighbor (kNN) search has wide applications in many areas, including data mining, machine learning, statistics and many applied domains. Inspired by the success of ensemble methods and the flexibility of tree-based methodology, we propose random projection forests (rpForests), for kNN search. rpForests finds kNNs by aggregating results from an ensemble of random projection trees with each constructed recursively through a series of carefully chosen random projections. rpForests achieves a remarkable accuracy in terms of fast decay in the missing rate of kNNs and that of discrepancy i…
Zhejiang University（浙江大学）；Central South University（中南大学）
Multi-label classification has attracted an increasing amount of attention in recent years. To this end, many algorithms have been developed to classify multi-label data in an effective manner. However, they usually do not consider the pairwise relations indicated by sample labels, which actually play important roles in multi-label classification. Inspired by this, we naturally extend the traditional pairwise constraints to the multi-label scenario via a flexible thresholding scheme. Moreover, to improve the generalization ability of the classifier, we adopt a boosting-like strategy to constru…
University of Trieste（的里雅斯特大学）；University of Edinburgh（爱丁堡大学）；SISSA（国际高等研究院）；Saarland University（萨尔兰大学）
We propose two training techniques for improving the robustness of Neural Networks to adversarial attacks, i.e. manipulations of the inputs that are maliciously crafted to fool networks into incorrect predictions. Both methods are independent of the chosen attack and leverage random projections of the original inputs, with the purpose of exploiting both dimensionality reduction and some characteristic geometrical properties of adversarial perturbations. The first technique is called RP-Ensemble and consists of an ensemble of networks trained on multiple projected versions of the original input…
Tongji University（同济大学）；University of Connecticut（康涅狄格大学）
Distributionally Robust Optimization (DRO), which aims to find an optimal decision that minimizes the worst case cost over the ambiguity set of probability distribution, has been widely applied in diverse applications, e.g., network behavior analysis, risk management, etc. However, existing DRO techniques face three key challenges: 1) how to deal with the asynchronous updating in a distributed environment; 2) how to leverage the prior distribution effectively; 3) how to properly adjust the degree of robustness according to different scenarios. To this end, we propose an asynchronous distribute…
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Effat University（Effat大学）
Datasets from the fields of bioinformatics, chemometrics, and face recognition are typically characterized by small samples of high-dimensional data. Among the many variants of linear discriminant analysis that have been proposed in order to rectify the issues associated with classification in such a setting, the classifier in [1], composed of an ensemble of randomly projected linear discriminants, seems especially promising; it is computationally efficient and, with the optimal projection dimension parameter setting, is competitive with the state-of-the-art. In this work, we seek to further u…
Distributional Determinantal Point Process for Repulsive Clustering of Distributions
Department of Statistics and Data Sciences, University of Texas at Austin（统计与数据科学系，德克萨斯大学奥斯汀分校）；Department of Mathematics, University of Texas at Austin（数学系，德克萨斯大学奥斯汀分校）；Department of Biostatistics and Informatics, University of Colorado Anschutz（生物统计学与信息学系，科罗拉多大学安舒茨分校）
Relative Wasserstein Angle and the Problem of the $W_2$-Nearest Gaussian Distribution
Department of Computer Science, George Washington University, Washington D.C., USA（计算机科学系，乔治·华盛顿大学，华盛顿特区，美国）；MAE, George Washington University, Washington D.C., USA（机械工程与应用科学系，乔治·华盛顿大学，华盛顿特区，美国）
Calibrated Multivariate Distributional Regression with Pre-Rank Regularization
Department of Statistics and Data Science, Mohamed Bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE（统计与数据科学系，Mohamed Bin Zayed人工智能大学，阿布扎比，阿联酋）
The ability to detect Out-of-Distribution (OOD) data is important in safety-critical applications of deep learning. The aim is to separate In-Distribution (ID) data drawn from the training distribution from OOD data using a measure of uncertainty extracted from a deep neural network. Deep Ensembles are a well-established method of improving the quality of uncertainty estimates produced by deep neural networks, and have been shown to have superior OOD detection performance compared to single models. An existing intuition in the literature is that the diversity of Deep Ensemble predictions indic…
