The empirical success of distributional reinforcement learning (RL) highly relies on the choice of distribution divergence equipped with an appropriate distribution representation. In this paper, we propose \textit{Sinkhorn distributional RL (SinkhornDRL)}, which leverages Sinkhorn divergence, a regularized Wasserstein loss, to minimize the difference between current and target Bellman return distributions. Theoretically, we prove the contraction properties of SinkhornDRL, aligning with the interpolation nature of Sinkhorn divergence between Wasserstein distance and Maximum Mean Discrepancy (MMD). The introduced SinkhornDRL enriches the family of distributional RL algorithms, contributing to interpreting the algorithm behaviors compared with existing approaches by our investigation into their relationships. Empirically, we show that SinkhornDRL consistently outperforms or matches existing algorithms on the Atari games suite and particularly stands out in the multi-dimensional reward setting. \thanks{Code is available in \url{https://github.com/datake/SinkhornDistRL}.}.
Wasserstein Policy Gradient for Entropy-Regularized Linear-Quadratic Control
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Wasserstein Distributionally Robust Regret Optimization for Reinforcement Learning from Human Feedback
Department of Statistics and Operations Research, University of North Carolina（统计与运筹学系，北卡罗来纳大学）；Imperial Business School, Imperial College London（帝国理工学院伦敦商学院）；Department of Management Science and Engineering, Stanford University（管理科学与工程系，斯坦福大学）
Well-Posed KL-Regularized Control via Wasserstein and Kalman-Wasserstein KL Divergences
Department of Mathematics, Technical University of Munich \& Munich Center for Machine Learning, Germany. The majority of the work was conducted while at the Institute of Mathematics at the Technical University of Berlin, Germany \& the Berlin Mathematical School.；Institute for Data Science Foundations, Hamburg University of Technology, Germany；Santa Fe Institute, USA
Quantile-Coupled Flow Matching for Distributional Reinforcement Learning
Oxford Robotics Institute, University of Oxford（牛津大学机器人研究所）；Bristol Robotics Laboratory, University of the West of England（布里斯托尔机器人实验室，西英格兰大学）；Dyson Institute of Engineering & Technology（戴森工程与技术研究院）
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；IBM Research（IBM研究院）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
Wasserstein barycenters provide a geometrically meaningful way to aggregate probability distributions, built on the theory of optimal transport. They are difficult to compute in practice, however, leading previous work to restrict their supports to finite sets of points. Leveraging a new dual formulation for the regularized Wasserstein barycenter problem, we introduce a stochastic algorithm that constructs a continuous approximation of the barycenter. We establish strong duality and use the corresponding primal-dual relationship to parametrize the barycenter implicitly using the dual potential…
We develop Distributionally Robust Optimization (DRO) formulations for Multivariate Linear Regression (MLR) and Multiclass Logistic Regression (MLG) when both the covariates and responses/labels may be contaminated by outliers. The DRO framework uses a probabilistic ambiguity set defined as a ball of distributions that are close to the empirical distribution of the training set in the sense of the Wasserstein metric. We relax the DRO formulation into a regularized learning problem whose regularizer is a norm of the coefficient matrix. We establish out-of-sample performance guarantees for the s…
In many applications in statistics and machine learning, the availability of data samples from multiple possibly heterogeneous sources has become increasingly prevalent. On the other hand, in distributionally robust optimization, we seek data-driven decisions which perform well under the most adverse distribution from a nominal distribution constructed from data samples within a certain discrepancy of probability distributions. However, it remains unclear how to achieve such distributional robustness in model learning and estimation when data samples from multiple sources are available. In thi…
The University of Sydney（悉尼大学）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）
We study a model for adversarial classification based on distributionally robust chance constraints. We show that under Wasserstein ambiguity, the model aims to minimize the conditional value-at-risk of the distance to misclassification, and we explore links to adversarial classification models proposed earlier and to maximum-margin classifiers. We also provide a reformulation of the distributionally robust model for linear classification, and show it is equivalent to minimizing a regularized ramp loss objective. Numerical experiments show that, despite the nonconvexity of this formulation, st…
Aalto University（阿尔托大学）；Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）
Gaussian distributions are plentiful in applications dealing in uncertainty quantification and diffusivity. They furthermore stand as important special cases for frameworks providing geometries for probability measures, as the resulting geometry on Gaussians is often expressible in closed-form under the frameworks. In this work, we study the Gaussian geometry under the entropy-regularized 2-Wasserstein distance, by providing closed-form solutions for the distance and interpolations between elements. Furthermore, we provide a fixed-point characterization of a population barycenter when restrict…
The University of Sydney Business School（悉尼大学商学院）；The University of Sydney（悉尼大学）
This paper introduces Wasserstein Adversarially Regularized Graph Autoencoder (WARGA), an implicit generative algorithm that directly regularizes the latent distribution of node embedding to a target distribution via the Wasserstein metric. The proposed method has been validated in tasks of link prediction and node clustering on real-world graphs, in which WARGA generally outperforms state-of-the-art models based on Kullback-Leibler (KL) divergence and typical adversarial framework.
VinAI Research（VinAI研究院）；University of Texas, Austin（德克萨斯大学奥斯汀分校）
Relational regularized autoencoder (RAE) is a framework to learn the distribution of data by minimizing a reconstruction loss together with a relational regularization on the latent space. A recent attempt to reduce the inner discrepancy between the prior and aggregated posterior distributions is to incorporate sliced fused Gromov-Wasserstein (SFG) between these distributions. That approach has a weakness since it treats every slicing direction similarly, meanwhile several directions are not useful for the discriminative task. To improve the discrepancy and consequently the relational regulari…
Wasserstein distributionally robust optimization (DRO) aims to find robust and generalizable solutions by hedging against data perturbations in Wasserstein distance. Despite its recent empirical success in operations research and machine learning, existing performance guarantees for generic loss functions are either overly conservative due to the curse of dimensionality, or plausible only in large sample asymptotics. In this paper, we develop a non-asymptotic framework for analyzing the out-of-sample performance for Wasserstein robust learning and the generalization bound for its related Lipsc…
Distributional Reduction: Unifying Dimensionality Reduction and Clustering with Gromov-Wasserstein
ENS de Lyon, CNRS, UMPA UMR 5669（里昂大学、国家科学研究中心、UMPA UMR 5669）；EPFL, LTS4（苏黎世联邦理工学院、LTS4）；Université Bretagne Sud, IRISA UMR 6074（布列塔尼南部大学、IRISA UMR 6074）；École polytechnique, Institut Polytechnique de Paris, CMAP UMR 7641（巴黎高等学院、巴黎理工学院、CMAP UMR 7641）；Inria, ENS de Lyon, CNRS, Université Claude Bernard Lyon 1, LIP UMR 5668（法国国家科学研究中心、里昂大学、 Claude Bernard Lyon 1 大学、LIP UMR 5668）
School of Computing, Australian National University（澳大利亚国立大学计算机学院）
We propose a learning framework for graph kernels, which is theoretically grounded on regularizing optimal transport. This framework provides a novel optimal transport distance metric, namely Regularized Wasserstein (RW) discrepancy, which can preserve both features and structure of graphs via Wasserstein distances on features and their local variations, local barycenters and global connectivity. Two strongly convex regularization terms are introduced to improve the learning ability. One is to relax an optimal alignment between graphs to be a cluster-to-cluster mapping between their locally co…
Distributional Matrix Completion via Nearest Neighbors in the Wasserstein Space
Industrial Engineering and Operations Research, Columbia University（哥伦比亚大学工业工程与运筹学系）；Statistics and Data Science, Cornell Tech, Cornell University（康奈尔大学数据科学与统计学系）；Operations Research and Information Engineering, Cornell Tech, Cornell University（康奈尔大学运筹学与信息工程系）
VinAI Research（VinAI研究院）；University of Texas, Austin（得克萨斯大学奥斯汀分校）
Sliced-Wasserstein distance (SW) and its variant, Max Sliced-Wasserstein distance (Max-SW), have been used widely in the recent years due to their fast computation and scalability even when the probability measures lie in a very high dimensional space. However, SW requires many unnecessary projection samples to approximate its value while Max-SW only uses the most important projection, which ignores the information of other useful directions. In order to account for these weaknesses, we propose a novel distance, named Distributional Sliced-Wasserstein distance (DSW), that finds an optimal dist…
Wasserstein multivariate auto-regressive models for modeling distributional time series
Université Grenoble Alpes, CNRS, Inria, Grenoble INP, LJK（格勒诺布尔阿尔卑斯大学、国家科学研究中心、法国国家信息与自动化研究所、格勒诺布尔INP、LJK研究中心）；Institut de Mathématiques de Bordeaux, Université de Bordeaux（波尔多数学研究所、波尔多大学）
RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）
This work studies the convergence and finite sample approximations of entropic regularized Wasserstein distances in the Hilbert space setting. Our first main result is that for Gaussian measures on an infinite-dimensional Hilbert space, convergence in the 2-Sinkhorn divergence is {\it strictly weaker} than convergence in the exact 2-Wasserstein distance. Specifically, a sequence of centered Gaussian measures converges in the 2-Sinkhorn divergence if the corresponding covariance operators converge in the Hilbert-Schmidt norm. This is in contrast to the previous known result that a sequence of c…
LIPN-CNRS（法国国家科学研究中心计算机科学实验室）；Université Sorbonne Paris Nord（索邦巴黎北大学）
This paper deals with a clustering algorithm for histogram data based on a Self-Organizing Map (SOM) learning. It combines a dimension reduction by SOM and the clustering of the data in a reduced space. Related to the kind of data, a suitable dissimilarity measure between distributions is introduced: the $L_2$ Wasserstein distance. Moreover, the number of clusters is not fixed in advance but it is automatically found according to a local data density estimation in the original space. Applications on synthetic and real data sets corroborate the proposed strategy.
The Chinese University of Hong Kong（香港中文大学）；Nanjing University（南京大学）
Wasserstein \textbf{D}istributionally \textbf{R}obust \textbf{O}ptimization (DRO) is concerned with finding decisions that perform well on data that are drawn from the worst-case probability distribution within a Wasserstein ball centered at a certain nominal distribution. In recent years, it has been shown that various DRO formulations of learning models admit tractable convex reformulations. However, most existing works propose to solve these convex reformulations by general-purpose solvers, which are not well-suited for tackling large-scale problems. In this paper, we focus on a family of W…
Tsinghua-Berkeley Shenzhen Institute, Tsinghua University（清华大学深圳国际研究生院（清华-伯克利深圳学院））；School of Industrial and Systems Engineering, Georgia Institute of Technology（佐治亚理工学院工业与系统工程系）
Given multiple source domains, domain generalization aims at learning a universal model that performs well on any unseen but related target domain. In this work, we focus on the domain generalization scenario where domain shifts occur among class-conditional distributions of different domains. Existing approaches are not sufficiently robust when the variation of conditional distributions given the same class is large. In this work, we extend the concept of distributional robust optimization to solve the class-conditional domain generalization problem. Our approach optimizes the worst-case perf…
Department of Mathematics, University of California, Los Angeles, CA 90095（加州大学洛杉矶分校数学系）；Department of Mathematics, University of South Carolina, Columbia, SC 29208（南卡罗来纳大学数学系）
Enhancing Distributional Robustness in Principal Component Analysis by Wasserstein Distances
The Hong Kong Polytechnic University（香港理工大学）；Chinese Academy of Sciences（中国科学院）；Academy of Mathematics and Systems Science（数学系统科学研究院）；University of Chinese Academy of Sciences（中国科学院大学）
Tsinghua University（清华大学）；University of Toronto（多伦多大学）
In prescriptive analytics, the decision-maker observes historical samples of $(X, Y)$, where $Y$ is the uncertain problem parameter and $X$ is the concurrent covariate, without knowing the joint distribution. Given an additional covariate observation $x$, the goal is to choose a decision $z$ conditional on this observation to minimize the cost $\mathbb{E}[c(z,Y)|X=x]$. This paper proposes a new distributionally robust approach under Wasserstein ambiguity sets, in which the nominal distribution of $Y|X=x$ is constructed based on the Nadaraya-Watson kernel estimator concerning the historical dat…
It is well-known that deep neural networks (DNNs) are susceptible to adversarial attacks, exposing a severe fragility of deep learning systems. As the result, adversarial training (AT) method, by incorporating adversarial examples during training, represents a natural and effective approach to strengthen the robustness of a DNN-based classifier. However, most AT-based methods, notably PGD-AT and TRADES, typically seek a pointwise adversary that generates the worst-case adversarial example by independently perturbing each data sample, as a way to "probe" the vulnerability of the classifier. Arg…
Sparse Transformer Architectures via Regularized Wasserstein Proximal Operator with $L_1$ Prior
Department of Mathematics, University of California, Los Angeles, CA, USA（加州大学洛杉矶分校数学系）；Department of Mathematics, University of South Carolina, Columbia, SC, USA（南卡罗来纳大学数学系）
Generalizing to Unseen Domains with Wasserstein Distributional Robustness under Limited Source Knowledge
Tsinghua-Berkeley Shenzhen Institute, Tsinghua University（清华大学-伯克利深圳学院，清华大学）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）；School of Industrial and Systems and Engineering, Georgia Institute of Technology（佐治亚理工学院工业与系统工程学院）
Accelerated Multiple Wasserstein Gradient Flows for Multi-objective Distributional Optimization
Graduate School of Information Science and Technology, Hokkaido University, Japan（信息科学与技术研究生学校，北海道大学，日本）；Bioinformatics Center, Kyoto University, Japan（生物信息学中心，京都大学，日本）；Institute of Information Technology, Vietnam Academy of Science and Technology, Vietnam（信息技术研究所，越南科学与技术学院，越南）
