This paper introduces a novel generative model for discrete distributions based on continuous normalizing flows on the submanifold of factorizing discrete measures. Integration of the flow gradually assigns categories and avoids issues of discretizing the latent continuous model like rounding, sample truncation etc. General non-factorizing discrete distributions capable of representing complex statistical dependencies of structured discrete data, can be approximated by embedding the submanifold into a the meta-simplex of all joint discrete distributions and data-driven averaging. Efficient training of the generative model is demonstrated by matching the flow of geodesics of factorizing discrete distributions. Various experiments underline the approach's broad applicability.
University of Science and Technology Beijing（北京科技大学）；Technical University of Denmark（丹麦技术大学）
Deep generative models are tremendously successful in learning low-dimensional latent representations that well-describe the data. These representations, however, tend to much distort relationships between points, i.e. pairwise distances tend to not reflect semantic similarities well. This renders unsupervised tasks, such as clustering, difficult when working with the latent representations. We demonstrate that taking the geometry of the generative model into account is sufficient to make simple clustering algorithms work well over latent representations. Leaning on the recent finding that dee…
The length of the geodesic between two data points along a Riemannian manifold, induced by a deep generative model, yields a principled measure of similarity. Current approaches are limited to low-dimensional latent spaces, due to the computational complexity of solving a non-convex optimisation problem. We propose finding shortest paths in a finite graph of samples from the aggregate approximate posterior, that can be solved exactly, at greatly reduced runtime, and without a notable loss in quality. Our approach, therefore, is hence applicable to high-dimensional problems, e.g., in the visual…
Big data Institute（大数据研究所）；School of Computer Science（计算机学院）；Shenzhen University（深圳大学）；Benha University（本哈大学）；Peng Cheng Laboratory（鹏城实验室）
Spectrum management and resource allocation (RA) problems are challenging and critical in a vast number of research areas such as wireless communications and computer networks. The traditional approaches for solving such problems usually consume time and memory, especially for large size problems. Recently different machine learning approaches have been considered as potential promising techniques for combinatorial optimization problems, especially the generative model of the deep neural networks. In this work, we propose a resource allocation deep autoencoder network, as one of the promising …
University of Science and Technology of China（中国科学技术大学）；Alibaba Inc.（阿里巴巴集团）
Noise injection has been proved to be one of the key technique advances in generating high-fidelity images. Despite its successful usage in GANs, the mechanism of its validity is still unclear. In this paper, we propose a geometric framework to theoretically analyze the role of noise injection in GANs. Based on Riemannian geometry, we successfully model the noise injection framework as fuzzy equivalence on the geodesic normal coordinates. Guided by our theories, we find that the existing method is incomplete and a new strategy for noise injection is devised. Experiments on image generation and…
Interpolating between points is a problem connected simultaneously with finding geodesics and study of generative models. In the case of geodesics, we search for the curves with the shortest length, while in the case of generative models we typically apply linear interpolation in the latent space. However, this interpolation uses implicitly the fact that Gaussian is unimodal. Thus the problem of interpolating in the case when the latent density is non-Gaussian is an open problem. In this paper, we present a general and unified approach to interpolation, which simultaneously allows us to search…
MPI Tübingen（图宾根马普研究所）；Microsoft Research Cambridge（微软剑桥研究院）；ETH Zürich（苏黎世联邦理工学院）
Variational Autoencoders (VAEs) are expressive latent variable models that can be used to learn complex probability distributions from training data. However, the quality of the resulting model crucially relies on the expressiveness of the inference model. We introduce Adversarial Variational Bayes (AVB), a technique for training Variational Autoencoders with arbitrarily expressive inference models. We achieve this by introducing an auxiliary discriminative network that allows to rephrase the maximum-likelihood-problem as a two-player game, hence establishing a principled connection between VA…
Generative Modeling of Discrete Data Using Geometric Latent Subspaces
Institute for Mathematics, Heidelberg University（海德堡大学数学研究所）；Mannheim Institute for Intelligent Systems in Medicine, Heidelberg University（海德堡大学曼海姆智能医学系统研究所）；IWR, Heidelberg University（海德堡大学IWR研究所）；Research Station Geometry and Dynamics, Heidelberg University（海德堡大学几何与动力学研究站）
Recent work has shown that deep generative models assign higher likelihood to out-of-distribution inputs than to training data. We show that a factor underlying this phenomenon is a mismatch between the nature of the prior distribution and that of the data distribution, a problem found in widely used deep generative models such as VAEs and Glow. While a typical choice for a prior distribution is a standard Gaussian distribution, properties of distributions of real data sets may not be consistent with a unimodal prior distribution. This paper focuses on the relationship between the choice of a …
Université Paris Dauphine - PSL Research University（巴黎第九大学-巴黎文理研究大学）；CEREMADE（CEREMADE（巴黎第九大学应用数学研究中心））
The quality of generative models (such as Generative adversarial networks and Variational Auto-Encoders) depends heavily on the choice of a good probability distance. However some popular metrics like the Wasserstein or the Sliced Wasserstein distances, the Jensen-Shannon divergence, the Kullback-Leibler divergence, lack convenient properties such as (geodesic) convexity, fast evaluation and so on. To address these shortcomings, we introduce a class of distances that have built-in convexity. We investigate the relationship with some known paradigms (sliced distances - a synonym for Radon dista…
Tsinghua University（清华大学）；University of California, Los Angeles（加州大学洛杉矶分校）；Peking University（北京大学）；Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）
Probabilistic Circuits (PCs) are a general and unified computational framework for tractable probabilistic models that support efficient computation of various inference tasks (e.g., computing marginal probabilities). Towards enabling such reasoning capabilities in complex real-world tasks, Liu et al. (2022) propose to distill knowledge (through latent variable assignments) from less tractable but more expressive deep generative models. However, it is still unclear what factors make this distillation work well. In this paper, we theoretically and empirically discover that the performance of a …
Quasi Monte Carlo methods enable extremely low-dimensional deep generative models
Department of Electrical and Computer Engineering, Duke University（杜克大学电气与计算机工程系）；Center for Cognitive Neuroscience, Duke University（杜克大学认知神经科学中心）；Center for Neural Science, New York University（纽约大学神经科学中心）；Center for Computational Neuroscience, Flatiron Institute（Flatiron研究所计算神经科学中心）
History Matching under Uncertainty of Geological Scenarios with Implicit Geological Realism Control with Generative Deep Learning and Graph Convolutions
Generative neural networks have a well recognized ability to estimate underlying manifold structure of high dimensional data. However, if a single latent space is used, it is not possible to faithfully represent a manifold with topology different from Euclidean space. In this work we define the general class of Atlas Generative Models (AGMs), models with hybrid discrete-continuous latent space that estimate an atlas on the underlying data manifold together with a partition of unity on the data space. We identify existing examples of models from various popular generative paradigms that fit int…
German Cancer Research Center (DKFZ)（德国癌症研究中心）；Heidelberg University（海德堡大学）；Helmholtz Information and Data Science School for Health（亥姆霍兹健康信息与数据科学学院）；Cancer Research UK Cambridge Institute, University of Cambridge（剑桥大学英国癌症研究中心剑桥研究所）
Photoacoustic tomography (PAT) has the potential to recover morphological and functional tissue properties with high spatial resolution. However, previous attempts to solve the optical inverse problem with supervised machine learning were hampered by the absence of labeled reference data. While this bottleneck has been tackled by simulating training data, the domain gap between real and simulated images remains an unsolved challenge. We propose a novel approach to PAT image synthesis that involves subdividing the challenge of generating plausible simulations into two disjoint problems: (1) Pro…
Attribution assignment for deep-generative sequence models enables interpretability analysis using positive-only data
Oslo University Hospital（奥斯陆大学医院）；University of Oslo（奥斯陆大学）；Johannes Kepler University Linz（约翰·开普勒林茨大学）；Novo Nordisk A/S（诺和诺德公司）；University Hospital Basel（巴塞尔大学医院）；Imprint Labs, LLC.（Imprint Labs有限责任公司）
Joint Model Assignment and Resource Allocation for Cost-Effective Mobile Generative Services
School of Electronic Information and Communications, Huazhong University of Science and Technology（华中科技大学电子信息与通信学院）；School of Computer Science and Engineering, Central South University（中南大学计算机科学与工程学院）；University of Windsor（温莎大学）
Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；University of Chinese Academy of Sciences（中国科学院大学）；School of Mathematics and Statistics, Fujian Normal University（福建师范大学数学与统计学学院）；Beijing Institute of Technology（北京理工大学）
This paper extends the recently introduced assignment flow approach for supervised image labeling to unsupervised scenarios where no labels are given. The resulting self-assignment flow takes a pairwise data affinity matrix as input data and maximizes the correlation with a low-rank matrix that is parametrized by the variables of the assignment flow, which entails an assignment of the data to themselves through the formation of latent labels (feature prototypes). A single user parameter, the neighborhood size for the geometric regularization of assignments, drives the entire process. By smooth…
University of Utah（犹他大学）；IBM Research AI（IBM人工智能研究院）
Deep generative models learn a mapping from a low dimensional latent space to a high-dimensional data space. Under certain regularity conditions, these models parameterize nonlinear manifolds in the data space. In this paper, we investigate the Riemannian geometry of these generated manifolds. First, we develop efficient algorithms for computing geodesic curves, which provide an intrinsic notion of distance between points on the manifold. Second, we develop an algorithm for parallel translation of a tangent vector along a path on the manifold. We show how parallel translation can be used to ge…
Exploiting the deep generative model's remarkable ability of learning the data-manifold structure, some recent researches proposed a geometric data interpolation method based on the geodesic curves on the learned data-manifold. However, this interpolation method often gives poor results due to a topological difference between the model and the dataset. The model defines a family of simply-connected manifolds, whereas the dataset generally contains disconnected regions or holes that make them non-simply-connected. To compensate this difference, we propose a novel density regularizer that make t…
DTU Compute, Technical University of Denmark, Kongens Lyngby, Denmark（丹麦技术大学计算机系）；Department of Computer Science, University of Virginia, Charlottesville (VA), USA（弗吉尼亚大学计算机科学系）；Department of Electrical and Computer Engineering, University of Virginia, Charlottesville (VA), USA（弗吉尼亚大学电气与计算机工程系）
Task Assignment and Exploration Optimization for Low Altitude UAV Rescue via Generative AI Enhanced Multi-agent Reinforcement Learning
Guangxi University Key Laboratory of Intelligent Networking and Scenario System (School of Information and Communication, Guilin University of Electronic Technology(GUET))（广西智能网络与场景系统重点实验室（信息与通信学院，桂林电子科技大学））；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算机与数据科学学院）；School of Architecture and Transportation Engineering, GUET（建筑与交通运输工程学院，GUET）；National Engineering Laboratory for Comprehensive Transportation Big Data Application Technology (Guangxi)（综合交通运输大数据应用技术国家工程实验室（广西））；Department of Informatics and Communication Engineering, Xiamen University（信息与通信工程系，厦门大学）；College of Computer Science and Technology, Jilin University（计算机科学与技术学院，吉林大学）；Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University（教育部符号计算与知识工程重点实验室，吉林大学）
University of Cambridge（剑桥大学）；The Alan Turing Institute（艾伦·图灵研究所）；University of Rome Tor Vergata（罗马第二大学）；Harvard Medical School（哈佛医学院）
Graphs are a natural representation of brain activity derived from functional magnetic imaging (fMRI) data. It is well known that clusters of anatomical brain regions, known as functional connectivity networks (FCNs), encode temporal relationships which can serve as useful biomarkers for understanding brain function and dysfunction. Previous works, however, ignore the temporal dynamics of the brain and focus on static graphs. In this paper, we propose a dynamic brain graph deep generative model (DBGDGM) which simultaneously clusters brain regions into temporally evolving communities and learns…
CrystalGRW: Generative Modeling of Crystal Structures with Targeted Properties via Geodesic Random Walks
Extreme Conditions Physics Research Laboratory（极端条件物理研究实验室）；Center of Excellence in Physics of Energy Materials (CE:PEM), Department of Physics, Faculty of Science, Chulalongkorn University, Bangkok 10330, Thailand（能源材料物理卓越中心，物理系，科学学院，朱拉隆梭大学，Bangkok 10330，泰国）；Thailand Center of Excellence in Physics, Ministry of Higher Education, Science, Research（泰国物理卓越中心，教育部，科学，研究）；Department of Chemistry（化学系）；Biochemistry, University of South Carolina, South Carolina 29208, USA（生物化学，南卡罗来纳大学，South Carolina 29208，美国）；Complex Systems Lab, Department of Physics, Faculty of Science, Chulalongkorn University, Bangkok 10330, Thailand（复杂系统实验室，物理系，科学学院，朱拉隆梭大学，Bangkok 10330，泰国）；Theoretical Physics Division, Department of Physics, Chemistry（理论物理 division，物理系，化学）
Element AI（Element AI公司）；MILA, Université de Montréal（蒙特利尔大学MILA研究所）；Stanford University（斯坦福大学）
Computing optimal transport maps between high-dimensional and continuous distributions is a challenging problem in optimal transport (OT). Generative adversarial networks (GANs) are powerful generative models which have been successfully applied to learn maps across high-dimensional domains. However, little is known about the nature of the map learned with a GAN objective. To address this problem, we propose a generative adversarial model in which the discriminator's objective is the $2$-Wasserstein metric. We show that during training, our generator follows the $W_2$-geodesic between the init…
Amazon Development Centre Scotland（亚马逊苏格兰开发中心）；University of Edinburgh（爱丁堡大学）；The Alan Turing Institute（艾伦·图灵研究所）
Capsule networks (see e.g. Hinton et al., 2018) aim to encode knowledge and reason about the relationship between an object and its parts. In this paper we specify a \emph{generative} model for such data, and derive a variational algorithm for inferring the transformation of each object and the assignments of observed parts to the objects. We apply this model to (i) data generated from multiple geometric objects like squares and triangles ("constellations"), and (ii) data from a parts-based model of faces. Recent work by Kosiorek et al. [2019] has used amortized inference via stacked capsule a…
Florida State University（佛罗里达州立大学）；University of South Florida（南佛罗里达大学）
Complex analyses involving multiple, dependent random quantities often lead to graphical models - a set of nodes denoting variables of interest, and corresponding edges denoting statistical interactions between nodes. To develop statistical analyses for graphical data, especially towards generative modeling, one needs mathematical representations and metrics for matching and comparing graphs, and subsequent tools, such as geodesics, means, and covariances. This paper utilizes a quotient structure to develop efficient algorithms for computing these quantities, leading to useful statistical tool…
