Probability density models based on deep networks have achieved remarkable success in modeling complex high-dimensional datasets. However, unlike kernel density estimators, modern neural models do not yield marginals or conditionals in closed form, as these quantities require the evaluation of seldom tractable integrals. In this work, we present the Marginalizable Density Model Approximator (MDMA), a novel deep network architecture which provides closed form expressions for the probabilities, marginals and conditionals of any subset of the variables. The MDMA learns deep scalar representations for each individual variable and combines them via learned hierarchical tensor decompositions into a tractable yet expressive CDF, from which marginals and conditional densities are easily obtained. We illustrate the advantage of exact marginalizability in several tasks that are out of reach of previous deep network-based density estimation models, such as estimating mutual information between arbitrary subsets of variables, inferring causality by testing for conditional independence, and inference with missing data without the need for data imputation, outperforming state-of-the-art models on these tasks. The model also allows for parallelized sampling with only a logarithmic dependence of the time complexity on the number of variables.
We introduce a flexible parametric mixed effects model for correlated binary data, with parameters that can be directly interpreted as marginal odds ratios. This leads to a robust estimation equation with an optimal weighting matrix being the inverse of a genuine model-based covariance matrix. Flexible correlation structures can be imposed by correlated random effects, and correlation parameters can be estimated by solving a composite likelihood score function. Marginal parameters are consistently estimated even when the conditional parametric model is misspecified, and the robust estimation p…
University College London（伦敦大学学院）；Facebook AI Research（脸书人工智能研究院）；University of Cambridge（剑桥大学）；University of California, Berkeley（加州大学伯克利分校）；Imperial College London（帝国理工学院）
Multi-marginal optimal transport enables one to compare multiple probability measures, which increasingly finds application in multi-task learning problems. One practical limitation of multi-marginal transport is computational scalability in the number of measures, samples and dimensionality. In this work, we propose a multi-marginal optimal transport paradigm based on random one-dimensional projections, whose (generalized) distance we term the sliced multi-marginal Wasserstein distance. To construct this distance, we introduce a characterization of the one-dimensional multi-marginal Kantorovi…
Our article addresses the problem of flexibly estimating a multivariate density while also attempting to estimate its marginals correctly. We do so by proposing two new estimators that try to capture the best features of mixture of normals and copula estimators while avoiding some of their weaknesses. The first estimator we propose is a mixture of normals copula model that is a flexible alternative to parametric copula models such as the normal and t copula. The second is a marginally adapted mixture of normals estimator that improves on the standard mixture of normals by using information con…
Distributionally Faithful Imputation via Positive Semi-Definite Kernel Density Estimation
Inria - Ecole Normale Supériore Paris（法国巴黎国家信息与自动化研究所-巴黎高等师范学院）；Department of Computer Science - UCL（伦敦大学学院计算机科学系）；SDA Bocconi School of Management（博科尼管理学院）
Some convergence results on the kernel density estimator are proven for a class of linear processes with cyclical effects. In particular we extend the results of Ho and Hsing (1996a) and Mielniczuk (1997) to the stationary processes for which the singularities of the spectral density are not limited to the origin. We show that the convergence rates and the limit distribution may be different in this context.
The ability to generate high-fidelity synthetic data is crucial when available (real) data is limited or where privacy and data protection standards allow only for limited use of the given data, e.g., in medical and financial data-sets. Current state-of-the-art methods for synthetic data generation are based on generative models, such as Generative Adversarial Networks (GANs). Even though GANs have achieved remarkable results in synthetic data generation, they are often challenging to interpret.Furthermore, GAN-based methods can suffer when used with mixed real and categorical variables.Moreov…
Copulas are popular as models for multivariate dependence because they allow the marginal densities and the joint dependence to be modeled separately. However, they usually require that the transformation from uniform marginals to the marginals of the joint dependence structure is known. This can only be done for a restricted set of copulas, e.g. a normal copula. Our article introduces copula-type estimators for flexible multivariate density estimation which also allow the marginal densities to be modeled separately from the joint dependence, as in copula modeling, but overcomes the lack of fl…
We consider distributions of ordered random vectors with given one-dimensional marginal distributions. We give an elementary necessary and sufficient condition for the existence of such a distribution with finite entropy. In this case, we give explicitly the density of the unique distribution which achieves the maximal entropy and compute the value of its entropy. This density is the unique one which has a product form on its support and the given one-dimensional marginals. The proof relies on the study of copulas with given one-dimensional marginal distributions for its order statistics.
We propose a user-friendly graphical tool, the half-disk density strip (HDDS), for visualizing and comparing probability density functions. The HDDS exploits color shading for representing a distribution in an intuitive way. In univariate settings, the half-disk density strip allows to immediately discern the key characteristics of a density, such as symmetry, dispersion, and multi-modality. In the multivariate settings, we define HDDS tables to generalize the concept of contingency tables. It is an array of half-disk density strips, which compactly displays the univariate marginal and conditi…
Hybrid Bernstein Normalizing Flows for Flexible Multivariate Density Regression with Interpretable Marginals
Institute for Applied Research HTWG - University of Applied Sciences（应用研究研究所 HTWG - 高等专业学院）；Chair of Statistics and Campus Institute Data Science (CIDAS) University of Göttingen（统计学教授及校园数据科学研究所 (CIDAS) 雷姆森大学）；Department of Statistics LMU Munich（统计学系 慕尼黑大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）
Sorbonne Université（索邦大学）；Université de Paris（巴黎大学）；University College London（伦敦大学学院）；Institut d’Astrophysique de Paris（巴黎天体物理研究所）；Flatiron Institute（弗莱提荣研究所）
High-dimensional probability density estimation for inference suffers from the "curse of dimensionality". For many physical inference problems, the full posterior distribution is unwieldy and seldom used in practice. Instead, we propose direct estimation of lower-dimensional marginal distributions, bypassing high-dimensional density estimation or high-dimensional Markov chain Monte Carlo (MCMC) sampling. By evaluating the two-dimensional marginal posteriors we can unveil the full-dimensional parameter covariance structure. We additionally propose constructing a simple hierarchy of fast neural …
Markov chain Monte Carlo (MCMC) is an established approach for uncertainty quantification and propagation in scientific applications. A key challenge in applying MCMC to scientific domains is computation: the target density of interest is often a function of expensive computations, such as a high-fidelity physical simulation, an intractable integral, or a slowly-converging iterative algorithm. Thus, using an MCMC algorithms with an expensive target density becomes impractical, as these expensive computations need to be evaluated at each iteration of the algorithm. In practice, these computatio…
Gaussian Belief Propagation (BP) algorithm is one of the most important distributed algorithms in signal processing and statistical learning involving Markov networks. It is well known that the algorithm correctly computes marginal density functions from a high dimensional joint density function over a Markov network in a finite number of iterations when the underlying Gaussian graph is acyclic. It is also known more recently that the algorithm produces correct marginal means asymptotically for cyclic Gaussian graphs under the condition of walk summability. This paper extends this convergence …
