Variational Autoencoders (VAEs) represent the given data in a low-dimensional latent space, which is generally assumed to be Euclidean. This assumption naturally leads to the common choice of a standard Gaussian prior over continuous latent variables. Recent work has, however, shown that this prior has a detrimental effect on model capacity, leading to subpar performance. We propose that the Euclidean assumption lies at the heart of this failure mode. To counter this, we assume a Riemannian structure over the latent space, which constitutes a more principled geometric view of the latent codes, and replace the standard Gaussian prior with a Riemannian Brownian motion prior. We propose an efficient inference scheme that does not rely on the unknown normalizing factor of this prior. Finally, we demonstrate that this prior significantly increases model capacity using only one additional scalar parameter.
Manifold-valued data naturally arises in medical imaging. In cognitive neuroscience, for instance, brain connectomes base the analysis of coactivation patterns between different brain regions on the analysis of the correlations of their functional Magnetic Resonance Imaging (fMRI) time series - an object thus constrained by construction to belong to the manifold of symmetric positive definite matrices. One of the challenges that naturally arises consists of finding a lower-dimensional subspace for representing such manifold-valued data. Traditional techniques, like principal component analysis…
Université Paris Cité（巴黎西岱大学）；INRIA（法国国家信息与自动化研究所）；Inserm（法国国家健康与医学研究院）；Centre de Recherche des Cordeliers（科德利埃研究中心）
This paper introduces a new interpretation of the Variational Autoencoder framework by taking a fully geometric point of view. We argue that vanilla VAE models unveil naturally a Riemannian structure in their latent space and that taking into consideration those geometrical aspects can lead to better interpolations and an improved generation procedure. This new proposed sampling method consists in sampling from the uniform distribution deriving intrinsically from the learned Riemannian latent space and we show that using this scheme can make a vanilla VAE competitive and even better than more …
A standard Variational Autoencoder, with a Euclidean latent space, is structurally incapable of capturing topological properties of certain datasets. To remove topological obstructions, we introduce Diffusion Variational Autoencoders with arbitrary manifolds as a latent space. A Diffusion Variational Autoencoder uses transition kernels of Brownian motion on the manifold. In particular, it uses properties of the Brownian motion to implement the reparametrization trick and fast approximations to the KL divergence. We show that the Diffusion Variational Autoencoder is capable of capturing topolog…
University of California, Santa Barbara（加州大学圣巴巴拉分校）
Recurrent Variational Autoencoder has been widely used for language modeling and text generation tasks. These models often face a difficult optimization problem, also known as the Kullback-Leibler (KL) term vanishing issue, where the posterior easily collapses to the prior, and the model will ignore latent codes in generative tasks. To address this problem, we introduce an improved Wasserstein Variational Autoencoder (WAE) with Riemannian Normalizing Flow (RNF) for text modeling. The RNF transforms a latent variable into a space that respects the geometric characteristics of input space, which…
Instituto Tecnologico Autonomo de Mexico（墨西哥自治技术学院）；Fox School of Business, Temple University（天普大学福克斯商学院）；Department of Mathematical Sciences, Durham University（杜伦大学数学科学系）
We provide a general framework for constructing probability distributions on Riemannian manifolds, taking advantage of area-preserving maps and isometries. Control over distributions' properties, such as parameters, symmetry and modality yield a family of flexible distributions that are straightforward to sample from, suitable for use within Monte Carlo algorithms and latent variable models, such as autoencoders. As an illustration, we empirically validate our approach by utilizing our proposed distributions within a variational autoencoder and a latent space network model. Finally, we take ad…
ETH Zürich（苏黎世联邦理工学院）；Massachusetts Institute of Technology（麻省理工学院）；Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology（麻省理工学院计算机科学与人工智能实验室）；Department of Computer Science, ETH Zürich（苏黎世联邦理工学院计算机科学系）；Computer Science and Artificial Intelligence Laboratory（计算机科学与人工智能实验室）；Department of Computer Science（计算机科学系）；MIT（麻省理工学院）；ETH（苏黎世联邦理工学院）
Euclidean geometry has historically been the typical "workhorse" for machine learning applications due to its power and simplicity. However, it has recently been shown that geometric spaces with constant non-zero curvature improve representations and performance on a variety of data types and downstream tasks. Consequently, generative models like Variational Autoencoders (VAEs) have been successfully generalized to elliptical and hyperbolic latent spaces. While these approaches work well on data with particular kinds of biases e.g. tree-like data for a hyperbolic VAE, there exists no generic a…
Université de Paris（巴黎大学）；Inria（法国国家信息与自动化研究所）；Centre de Recherche des Cordeliers（科德利埃研究中心）；Inserm（法国国家健康与医学研究院）；Sorbonne Université（索邦大学）；Institut du Cerveau - Paris Brain Institute（巴黎脑研究所）；CNRS（法国国家科学研究中心）；AP-HP Hôpital de la Pitié Salpêtrière（巴黎公立医院集团-萨伯特慈善医院）；Inria Ara（法国国家信息与自动化研究所Ara团队）
In this paper, we propose a new method to perform data augmentation in a reliable way in the High Dimensional Low Sample Size (HDLSS) setting using a geometry-based variational autoencoder. Our approach combines a proper latent space modeling of the VAE seen as a Riemannian manifold with a new generation scheme which produces more meaningful samples especially in the context of small data sets. The proposed method is tested through a wide experimental study where its robustness to data sets, classifiers and training samples size is stressed. It is also validated on a medical imaging classifica…
Bosch Center for Artificial Intelligence (BCAI)（博世人工智能中心）；Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）；Technical University of Denmark (DTU)（丹麦技术大学）；Max Planck Institute for Intelligent Systems (MPI-IS)（马克斯·普朗克智能系统研究所）
For robots to work alongside humans and perform in unstructured environments, they must learn new motion skills and adapt them to unseen situations on the fly. This demands learning models that capture relevant motion patterns, while offering enough flexibility to adapt the encoded skills to new requirements, such as dynamic obstacle avoidance. We introduce a Riemannian manifold perspective on this problem, and propose to learn a Riemannian manifold from human demonstrations on which geodesics are natural motion skills. We realize this with a variational autoencoder (VAE) over the space of pos…
Riemannian AmbientFlow: Towards Simultaneous Manifold Learning and Generative Modeling from Corrupted Data
Department of Mathematics, University of California, Los Angeles（数学系，加州大学洛杉矶分校）；Department of Statistics and Data Science, University of California, Los Angeles（统计与数据科学系，加州大学洛杉矶分校）
Deep neural networks often suffer from overconfidence which can be partly remedied by improved out-of-distribution detection. For this purpose, we propose a novel approach that allows for the generation of out-of-distribution datasets based on a given in-distribution dataset. This new dataset can then be used to improve out-of-distribution detection for the given dataset and machine learning task at hand. The samples in this dataset are with respect to the feature space close to the in-distribution dataset and therefore realistic and plausible. Hence, this dataset can also be used to safeguard…
Radial Compensation: The Inverse Base-Distribution Problem for Chart-Based Generative Models on Riemannian Manifolds
Human Nature Lab, Yale University（耶鲁大学人类自然实验室）；Department of Statistics and Data Science, The Wharton School, University of Pennsylvania（宾夕法尼亚大学统计与数据科学系、沃顿商学院）
Geodesic Calculus on Implicitly Defined Latent Manifolds
University of Bonn, Germany（波恩大学，德国）；Université Paris–Saclay, ENS Paris–Saclay, Centre Borelli, France（巴黎-萨克莱大学，巴黎-萨克莱高等师范学院，Borelli研究中心，法国）；University of Münster, Germany（穆尔斯特大学，德国）
Università della Svizzera italiana（意大利瑞士大学）；University of Manitoba（曼尼托巴大学）；University of Exeter（埃克塞特大学）；Politecnico di Milano（米兰理工大学）
Constant-curvature Riemannian manifolds (CCMs) have been shown to be ideal embedding spaces in many application domains, as their non-Euclidean geometry can naturally account for some relevant properties of data, like hierarchy and circularity. In this work, we introduce the CCM adversarial autoencoder (CCM-AAE), a probabilistic generative model trained to represent a data distribution on a CCM. Our method works by matching the aggregated posterior of the CCM-AAE with a probability distribution defined on a CCM, so that the encoder implicitly learns to represent data on the CCM to fool the dis…
