On the benefits of defining vicinal distributions in latent space
IIT Hyderabad（印度理工学院海得拉巴分校）
Puneet Mangla, Vedant Singh, Shreyas Jayant Havaldar, Vineeth N Balasubramanian
备注：
Accepted at Elsevier Pattern Recognition Letters (2021), Best Paper Award at CVPR 2021 Workshop on Adversarial Machine Learning in Real-World Computer Vision (AML-CV), Also accepted at ICLR 2021 Workshops on Robust-Reliable Machine Learning (Oral) and Generalization beyond the training distribution (Abstract)
The vicinal risk minimization (VRM) principle is an empirical risk minimization (ERM) variant that replaces Dirac masses with vicinal functions. There is strong numerical and theoretical evidence showing that VRM outperforms ERM in terms of generalization if appropriate vicinal functions are chosen. Mixup Training (MT), a popular choice of vicinal distribution, improves the generalization performance of models by introducing globally linear behavior in between training examples. Apart from generalization, recent works have shown that mixup trained models are relatively robust to input perturbations/corruptions and at the same time are calibrated better than their non-mixup counterparts. In this work, we investigate the benefits of defining these vicinal distributions like mixup in latent space of generative models rather than in input space itself. We propose a new approach - \textit{VarMixup (Variational Mixup)} - to better sample mixup images by using the latent manifold underlying the data. Our empirical studies on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate that models trained by performing mixup in the latent manifold learned by VAEs are inherently more robust to various input corruptions/perturbations, are significantly better calibrated, and exhibit more local-linear loss landscapes.
We address the problem of one-to-many mappings in supervised learning, where a single instance has many different solutions of possibly equal cost. The framework of conditional variational autoencoders describes a class of methods to tackle such structured-prediction tasks by means of latent variables. We propose to incentivise informative latent representations for increasing the generalisation capacity of conditional variational autoencoders. To this end, we modify the latent variable model by defining the likelihood as a function of the latent variable only and introduce an expressive multi…
Learning representations that disentangle the underlying factors of variability in data is an intuitive way to achieve generalization in deep models. In this work, we address the scenario where generative factors present a multimodal distribution due to the existence of class distinction in the data. We propose N-VAE, a model which is capable of separating factors of variation which are exclusive to certain classes from factors that are shared among classes. This model implements an explicitly compositional latent variable structure by defining a class-conditioned latent space and a shared lat…
Machine Learning Research Lab, Volkswagen Group（大众集团机器学习研究实验室）；Autonomous Intelligent Driving GmbH（自主智能驾驶有限公司）
Measuring the similarity between data points often requires domain knowledge, which can in parts be compensated by relying on unsupervised methods such as latent-variable models, where similarity/distance is estimated in a more compact latent space. Prevalent is the use of the Euclidean metric, which has the drawback of ignoring information about similarity of data stored in the decoder, as captured by the framework of Riemannian geometry. We propose an extension to the framework of variational auto-encoders allows learning flat latent manifolds, where the Euclidean metric is a proxy for the s…
The Australian National University（澳大利亚国立大学）；Australian Centre for Robotic Vision（澳大利亚机器人视觉中心）
Controllable image synthesis with user scribbles has gained huge public interest with the recent advent of text-conditioned latent diffusion models. The user scribbles control the color composition while the text prompt provides control over the overall image semantics. However, we note that prior works in this direction suffer from an intrinsic domain shift problem, wherein the generated outputs often lack details and resemble simplistic representations of the target domain. In this paper, we propose a novel guided image synthesis framework, which addresses this problem by modeling the output…
The goal of item response theoretic (IRT) models is to provide estimates of latent traits from binary observed indicators and at the same time to learn the item response functions (IRFs) that map from latent trait to observed response. However, in many cases observed behavior can deviate significantly from the parametric assumptions of traditional IRT models. Nonparametric IRT models overcome these challenges by relaxing assumptions about the form of the IRFs, but standard tools are unable to simultaneously estimate flexible IRFs and recover ability estimates for respondents. We propose a Baye…
Conventional uncertainty quantification methods usually lacks the capability of dealing with high-dimensional problems due to the curse of dimensionality. This paper presents a semi-supervised learning framework for dimension reduction and reliability analysis. An autoencoder is first adopted for mapping the high-dimensional space into a low-dimensional latent space, which contains a distinguishable failure surface. Then a deep feedforward neural network (DFN) is utilized to learn the mapping relationship and reconstruct the latent space, while the Gaussian process (GP) modeling technique is u…
HRL Laboratories, LLC（HRL实验室有限责任公司）；University of Virginia（弗吉尼亚大学）
In this paper we study generative modeling via autoencoders while using the elegant geometric properties of the optimal transport (OT) problem and the Wasserstein distances. We introduce Sliced-Wasserstein Autoencoders (SWAE), which are generative models that enable one to shape the distribution of the latent space into any samplable probability distribution without the need for training an adversarial network or defining a closed-form for the distribution. In short, we regularize the autoencoder loss with the sliced-Wasserstein distance between the distribution of the encoded training samples…
Washington University in St Louis（圣路易斯华盛顿大学）；Department of Neuroscience（神经科学系）
Generative adversarial networks (GANs) have emerged as a powerful unsupervised method to model the statistical patterns of real-world data sets, such as natural images. These networks are trained to map random inputs in their latent space to new samples representative of the learned data. However, the structure of the latent space is hard to intuit due to its high dimensionality and the non-linearity of the generator, which limits the usefulness of the models. Understanding the latent space requires a way to identify input codes for existing real-world images (inversion), and a way to identify…
Uncertainty-aware damage identification in short-span bridges via physics-informed variational autoencoder
Basque Center for Applied Mathematics (BCAM)（巴斯克应用数学中心）；Departamento de Matematica Aplicada, Universidad de Zaragoza/IUMA（萨拉戈萨大学应用数学系/数学及其应用研究所）；TECNALIA, Basque Research and Technology Alliance (BRTA)（特克纳利亚研究与创新中心，巴斯克科学与技术联盟）；Department of Mathematics, University of the Basque Country (UPV/EHU)（巴斯克大学数学系）
University of Edinburgh（爱丁堡大学）；University of Amsterdam（阿姆斯特丹大学）
Opinion summarization is the task of automatically creating summaries that reflect subjective information expressed in multiple documents, such as product reviews. While the majority of previous work has focused on the extractive setting, i.e., selecting fragments from input reviews to produce a summary, we let the model generate novel sentences and hence produce abstractive summaries. Recent progress in summarization has seen the development of supervised models which rely on large quantities of document-summary pairs. Since such training data is expensive to acquire, we instead consider the …
Eindhoven University of Technology（埃因霍温理工大学）；Khoury College of Computer Sciences（胡里计算机科学学院）；Northeastern University（东北大学）；University of Amsterdam（阿姆斯特丹大学）
Volkswagen Group of America Innovation Center California（大众汽车美国创新中心加州）；BayzAI
The lossless data compression algorithm based on Bayesian Attention Networks is derived from first principles. Bayesian Attention Networks are defined by introducing an attention factor per a training sample loss as a function of two sample inputs, from training sample and prediction sample. By using a sharpened Jensen's inequality we show that the attention factor is completely defined by a correlation function of the two samples w.r.t. the model weights. Due to the attention factor the solution for a prediction sample is mostly defined by a few training samples that are correlated with the p…
Linear modal analysis is a useful and effective tool for the design and analysis of structures. However, a comprehensive basis for nonlinear modal analysis remains to be developed. In the current work, a machine learning scheme is proposed with a view to performing nonlinear modal analysis. The scheme is focussed on defining a one-to-one mapping from a latent `modal' space to the natural coordinate space, whilst also imposing orthogonality of the mode shapes. The mapping is achieved via the use of the recently-developed cycle-consistent generative adversarial network (cycle-GAN) and an assembl…
We introduce a new category of multivariate conditional generative models and demonstrate its performance and versatility in probabilistic time series forecasting and simulation. Specifically, the output of quantile regression networks is expanded from a set of fixed quantiles to the whole Quantile Function by a univariate mapping from a latent uniform distribution to the target distribution. Then the multivariate case is solved by learning such quantile functions for each dimension's marginal distribution, followed by estimating a conditional Copula to associate these latent uniform random va…
Data-driven identification of latent port-Hamiltonian systems
Institute of Engineering and Computational Mechanics, University of Stuttgart（斯图加特大学工程与计算力学研究所）；Institute of Applied Mechanics – Data Analytics in Engineering, University of Stuttgart（斯图加特大学应用力学研究所（工程数据分析方向））；University of Twente（特文特大学）；Institute of Applied Analysis and Numerical Simulation, University of Stuttgart（斯图加特大学应用分析与数值模拟研究所）
A defining feature of sampling-based motion planning is the reliance on an implicit representation of the state space, which is enabled by a set of probing samples. Traditionally, these samples are drawn either probabilistically or deterministically to uniformly cover the state space. Yet, the motion of many robotic systems is often restricted to "small" regions of the state space, due to, for example, differential constraints or collision-avoidance constraints. To accelerate the planning process, it is thus desirable to devise non-uniform sampling strategies that favor sampling in those regio…
GTS: Inference-Time Scaling of Latent Reasoning with a Learnable Gaussian Thought Sampler
Department of Data Science & AI, Monash University（墨尔本大学数据科学与人工智能系）；Faculty of Medicine Dentistry and Health Sciences, University of Melbourne（墨尔本大学医学牙科与健康科学学院）；Department of Computer Science, University College London（伦敦大学学院计算机科学系）
LUCoS: Latent Unsupervised Context Selection for Tabular Foundation Models
Andalusian Research Institute in Data Science and Computational Intelligence (DaSCI)（安达卢西亚数据科学与计算智能研究 institute）；Department of Computer Science and Artificial Intelligence (DECSAI)（计算机科学与人工智能系）；Research Center in Information and Communication Technologies (CITIC)（信息与通信技术研究中心）；Instituto de Investigación Biosanitaria Ibs.GRANADA, University of Granada, Granada, 18071, Spain（格拉纳达大学生物医学研究所）
Hunting for "Oddballs" with Machine Learning: Detecting Anomalous Exoplanets Using a Deep-Learned Low-Dimensional Representation of Transit Spectra with Autoencoders
Department of Physics and Astronomy, University of Alabama, Tuscaloosa, AL, 35487, USA（物理与天文学系，阿拉巴马大学，塔斯卡洛萨，AL，35487，USA）；Physics Department, University of Florida, Gainesville, FL 32653, USA（物理系，佛罗里达大学，盖恩斯维尔，FL，32653，USA）
