Deep Gaussian Processes (DGPs) were proposed as an expressive Bayesian model capable of a mathematically grounded estimation of uncertainty. The expressivity of DPGs results from not only the compositional character but the distribution propagation within the hierarchy. Recently, [1] pointed out that the hierarchical structure of DGP well suited modeling the multi-fidelity regression, in which one is provided sparse observations with high precision and plenty of low fidelity observations. We propose the conditional DGP model in which the latent GPs are directly supported by the fixed lower fidelity data. Then the moment matching method in [2] is applied to approximate the marginal prior of conditional DGP with a GP. The obtained effective kernels are implicit functions of the lower-fidelity data, manifesting the expressivity contributed by distribution propagation within the hierarchy. The hyperparameters are learned via optimizing the approximate marginal likelihood. Experiments with synthetic and high dimensional data show comparable performance against other multi-fidelity regression methods, variational inference, and multi-output GP. We conclude that, with the low fidelity data and the hierarchical DGP structure, the effective kernel encodes the inductive bias for true function allowing the compositional freedom discussed in [3,4].
Universality of Gaussian-Mixture Reverse Kernels in Conditional Diffusion
Center for Mathematics and Interdisciplinary Sciences, Fudan University（复旦大学数学与交叉科学中心）；Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS)（上海数学与交叉科学研究院（SIMIS））；Northeastern University（东北大学）；Infectious Diseases Division, icddr,b（icddr,b传染病部门）；Rutgers University（罗格斯大学）
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Conditional independence testing is an important problem, especially in Bayesian network learning and causal discovery. Due to the curse of dimensionality, testing for conditional independence of continuous variables is particularly challenging. We propose a Kernel-based Conditional Independence test (KCI-test), by constructing an appropriate test statistic and deriving its asymptotic distribution under the null hypothesis of conditional independence. The proposed method is computationally efficient and easy to implement. Experimental results show that it outperforms other methods, especially …
Zalando Research（Zalando研究部）；Zalando SE（Zalando股份公司）；Freie Universität Berlin（柏林自由大学）；MPI for Intelligent Systems（智能系统马克斯·普朗克研究所）
We introduce a novel conditional density estimation model termed the conditional density operator (CDO). It naturally captures multivariate, multimodal output densities and shows performance that is competitive with recent neural conditional density models and Gaussian processes. The proposed model is based on a novel approach to the reconstruction of probability densities from their kernel mean embeddings by drawing connections to estimation of Radon-Nikodym derivatives in the reproducing kernel Hilbert space (RKHS). We prove finite sample bounds for the estimation error in a standard density…
In practical Bayesian optimization, we must often search over structures with differing numbers of parameters. For instance, we may wish to search over neural network architectures with an unknown number of layers. To relate performance data gathered for different architectures, we define a new kernel for conditional parameter spaces that explicitly includes information about which parameters are relevant in a given structure. We show that this kernel improves model quality and Bayesian optimization results over several simpler baseline kernels.
The University of Sydney（悉尼大学）；CSIRO（澳大利亚联邦科学与工业研究组织）；NVIDIA（英伟达）
In likelihood-free settings where likelihood evaluations are intractable, approximate Bayesian computation (ABC) addresses the formidable inference task to discover plausible parameters of simulation programs that explain the observations. However, they demand large quantities of simulation calls. Critically, hyperparameters that determine measures of simulation discrepancy crucially balance inference accuracy and sample efficiency, yet are difficult to tune. In this paper, we present kernel embedding likelihood-free inference (KELFI), a holistic framework that automatically learns model hyper…
University of Sydney（悉尼大学）；CSIRO（澳大利亚联邦科学与工业研究组织）；NVIDIA（英伟达）
Conditional kernel mean embeddings form an attractive nonparametric framework for representing conditional means of functions, describing the observation processes for many complex models. However, the recovery of the original underlying function of interest whose conditional mean was observed is a challenging inference task. We formalize deconditional kernel mean embeddings as a solution to this inverse problem, and show that it can be naturally viewed as a nonparametric Bayes' rule. Critically, we introduce the notion of task transformed Gaussian processes and establish deconditional kernel …
London School of Economics（伦敦政治经济学院）；ETH Zurich（苏黎世联邦理工学院）
Causal Optimal Transport (COT) results from imposing a temporal causality constraint on classic optimal transport problems, which naturally generates a new concept of distances between distributions on path spaces. The first application of the COT theory for sequential learning was given in Xu et al. (2020), where COT-GAN was introduced as an adversarial algorithm to train implicit generative models optimized for producing sequential data. Relying on (Xu et al., 2020), the contribution of the present paper is twofold. First, we develop a conditional version of COT-GAN suitable for sequence pre…
Rutgers University（罗格斯大学）；Institute for Advanced Studies（高等研究院）
It is desirable to combine the expressive power of deep learning with Gaussian Process (GP) in one expressive Bayesian learning model. Deep kernel learning showed success in adopting a deep network for feature extraction followed by a GP used as function model. Recently,it was suggested that, albeit training with marginal likelihood, the deterministic nature of feature extractor might lead to overfitting while the replacement with a Bayesian network seemed to cure it. Here, we propose the conditional Deep Gaussian Process (DGP) in which the intermediate GPs in hierarchical composition are supp…
Brown University（布朗大学）；National ICT Australia and ANU（澳大利亚国家信息通信技术中心与澳大利亚国立大学）
In this paper we de ne conditional random elds in reproducing kernel Hilbert spaces and show connections to Gaussian Process classi cation. More speci cally, we prove decomposition results for undirected graphical models and we give constructions for kernels. Finally we present e cient means of solving the optimization problem using reduced rank decompositions and we show how stationarity can be exploited e ciently in the optimization process.
Deep generative models are becoming a cornerstone of modern machine learning. Recent work on conditional generative adversarial networks has shown that learning complex, high-dimensional distributions over natural images is within reach. While the latest models are able to generate high-fidelity, diverse natural images at high resolution, they rely on a vast quantity of labeled data. In this work we demonstrate how one can benefit from recent work on self- and semi-supervised learning to outperform the state of the art on both unsupervised ImageNet synthesis, as well as in the conditional sett…
Bifidelity Parameter Estimation Using Conditional Diffusion Models
UT-Battelle, LLC（UT-巴特勒公司）；US Department of Energy（美国能源部）；The Ohio State University（俄亥俄州立大学）；Oak Ridge National Laboratory（橡树岭国家实验室）；Oak Ridge（橡树岭）；Computer Science and Mathematics Division（计算机科学与数学 division）
Signature Kernel Conditional Independence Tests in Causal Discovery for Stochastic Processes
Technical University of Munich（慕尼黑工业大学）；Helmholtz Munich（慕尼黑亥姆霍兹中心）；Munich Center for Machine Learning（慕尼黑机器学习中心）；University of Oxford（牛津大学）；Lund University（隆德大学）；Imperial College London（伦敦帝国学院）
School of Computing Science（计算科学学院）；Google（谷歌）；Simon Fraser University（西蒙弗雷泽大学）
A persistent challenge in conditional image synthesis has been to generate diverse output images from the same input image despite only one output image being observed per input image. GAN-based methods are prone to mode collapse, which leads to low diversity. To get around this, we leverage Implicit Maximum Likelihood Estimation (IMLE) which can overcome mode collapse fundamentally. IMLE uses the same generator as GANs but trains it with a different, non-adversarial objective which ensures each observed image has a generated sample nearby. Unfortunately, to generate high-fidelity images, prio…
Reasoning based on causality, instead of association has been considered as a key ingredient towards real machine intelligence. However, it is a challenging task to infer causal relationship/structure among variables. In recent years, an Independent Mechanism (IM) principle was proposed, stating that the mechanism generating the cause and the one mapping the cause to the effect are independent. As the conjecture, it is argued that in the causal direction, the conditional distributions instantiated at different value of the conditioning variable have less variation than the anti-causal directio…
Utrecht University（乌得勒支大学）；Georgia Institute of Technology（佐治亚理工学院）；Microsoft（微软）
Obtaining samples from the posterior distribution of inverse problems with expensive forward operators is challenging especially when the unknowns involve the strongly heterogeneous Earth. To meet these challenges, we propose a preconditioning scheme involving a conditional normalizing flow (NF) capable of sampling from a low-fidelity posterior distribution directly. This conditional NF is used to speed up the training of the high-fidelity objective involving minimization of the Kullback-Leibler divergence between the predicted and the desired high-fidelity posterior density for indirect measu…
HistDiT: A Structure-Aware Latent Conditional Diffusion Model for High-Fidelity Virtual Staining in Histopathology
Edge Hill University（埃奇希尔大学）；University of Nottingham Malaysia（诺丁汉大学马来西亚分校）；University of Southampton Malaysia（南安普顿大学马来西亚分校）；Cancer Research Malaysia（马来西亚癌症研究中心）
We present an operator-free, measure-theoretic approach to the conditional mean embedding (CME) as a random variable taking values in a reproducing kernel Hilbert space. While the kernel mean embedding of unconditional distributions has been defined rigorously, the existing operator-based approach of the conditional version depends on stringent assumptions that hinder its analysis. We overcome this limitation via a measure-theoretic treatment of CMEs. We derive a natural regression interpretation to obtain empirical estimates, and provide a thorough theoretical analysis thereof, including univ…
University of Sydney（悉尼大学）；Data61, CSIRO（联邦科学与工业研究组织数据61研究所）；Australian National University（澳大利亚国立大学）
Conditional kernel mean embeddings are nonparametric models that encode conditional expectations in a reproducing kernel Hilbert space. While they provide a flexible and powerful framework for probabilistic inference, their performance is highly dependent on the choice of kernel and regularization hyperparameters. Nevertheless, current hyperparameter tuning methods predominantly rely on expensive cross validation or heuristics that is not optimized for the inference task. For conditional kernel mean embeddings with categorical targets and arbitrary inputs, we propose a hyperparameter learning …
Current meta-learning approaches focus on learning functional representations of relationships between variables, i.e. on estimating conditional expectations in regression. In many applications, however, we are faced with conditional distributions which cannot be meaningfully summarized using expectation only (due to e.g. multimodality). Hence, we consider the problem of conditional density estimation in the meta-learning setting. We introduce a novel technique for meta-learning which combines neural representation and noise-contrastive estimation with the established literature of conditional…
Much of machine learning relies on comparing distributions with discrepancy measures. Stein's method creates discrepancy measures between two distributions that require only the unnormalized density of one and samples from the other. Stein discrepancies can be combined with kernels to define kernelized Stein discrepancies (KSDs). While kernels make Stein discrepancies tractable, they pose several challenges in high dimensions. We introduce kernelized complete conditional Stein discrepancies (KCC-SDs). Complete conditionals turn a multivariate distribution into multiple univariate distributions…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Technion, Israel Institute of Technology（以色列理工学院）
We propose to analyse the conditional distributional treatment effect (CoDiTE), which, in contrast to the more common conditional average treatment effect (CATE), is designed to encode a treatment's distributional aspects beyond the mean. We first introduce a formal definition of the CoDiTE associated with a distance function between probability measures. Then we discuss the CoDiTE associated with the maximum mean discrepancy via kernel conditional mean embeddings, which, coupled with a hypothesis test, tells us whether there is any conditional distributional effect of the treatment. Finally, …
CREST（法国 Crest 研究中心）；ENSAE（法国国家统计与经济管理学院）；Facebook AI Research（脸书人工智能研究院）；Université Paris-Dauphine（巴黎多芬大学）；Technion（以色列理工学院）
We propose a new conditional dependence measure and a statistical test for conditional independence. The measure is based on the difference between analytic kernel embeddings of two well-suited distributions evaluated at a finite set of locations. We obtain its asymptotic distribution under the null hypothesis of conditional independence and design a consistent statistical test from it. We conduct a series of experiments showing that our new test outperforms state-of-the-art methods both in terms of type-I and type-II errors even in the high dimensional setting.
We present a probabilistic deep learning methodology that enables the construction of predictive data-driven surrogates for stochastic systems. Leveraging recent advances in variational inference with implicit distributions, we put forth a statistical inference framework that enables the end-to-end training of surrogate models on paired input-output observations that may be stochastic in nature, originate from different information sources of variable fidelity, or be corrupted by complex noise processes. The resulting surrogates can accommodate high-dimensional inputs and outputs and are able …
SZTAKI: Institute for Computer Science and Control（SZTAKI：计算机科学与控制研究所）；Eötvös Loránd Research Network（厄特沃什·罗兰研究网络）
In this paper we suggest two statistical hypothesis tests for the regression function of binary classification based on conditional kernel mean embeddings. The regression function is a fundamental object in classification as it determines both the Bayes optimal classifier and the misclassification probabilities. A resampling based framework is presented and combined with consistent point estimators of the conditional kernel mean map, in order to construct distribution-free hypothesis tests. These tests are introduced in a flexible manner allowing us to control the exact probability of type I e…
Graduate University of Advanced Studies (SOKENDAI)（高等研究生院（SOKENDAI））；The Institute of Statistical Mathematics（统计数学研究所）；The University of Adelaide（阿德莱德大学）
University of Pittsburgh School of Medicine（匹兹堡大学医学院）
Developing feature selection algorithms that move beyond a pure correlational to a more causal analysis of observational data is an important problem in the sciences. Several algorithms attempt to do so by discovering the Markov blanket of a target, but they all contain a forward selection step which variables must pass in order to be included in the conditioning set. As a result, these algorithms may not consider all possible conditional multivariate combinations. We improve on this limitation by proposing a backward elimination method that uses a kernel-based conditional dependence measure t…
We propose a method for feature selection that employs kernel-based measures of independence to find a subset of covariates that is maximally predictive of the response. Building on past work in kernel dimension reduction, we show how to perform feature selection via a constrained optimization problem involving the trace of the conditional covariance operator. We prove various consistency results for this procedure, and also demonstrate that our method compares favorably with other state-of-the-art algorithms on a variety of synthetic and real data sets.
MPI for Intelligent Systems（马克斯·普朗克智能系统研究所）；Google Research（谷歌研究院）
We propose a new family of specification tests called kernel conditional moment (KCM) tests. Our tests are built on a novel representation of conditional moment restrictions in a reproducing kernel Hilbert space (RKHS) called conditional moment embedding (CMME). After transforming the conditional moment restrictions into a continuum of unconditional counterparts, the test statistic is defined as the maximum moment restriction (MMR) within the unit ball of the RKHS. We show that the MMR not only fully characterizes the original conditional moment restrictions, leading to consistency in both hyp…
Carnegie Mellon University（卡内基梅隆大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Conditional contrastive learning frameworks consider the conditional sampling procedure that constructs positive or negative data pairs conditioned on specific variables. Fair contrastive learning constructs negative pairs, for example, from the same gender (conditioning on sensitive information), which in turn reduces undesirable information from the learned representations; weakly supervised contrastive learning constructs positive pairs with similar annotative attributes (conditioning on auxiliary information), which in turn are incorporated into the representations. Although conditional co…
Kernel conditional tests from learning-theoretic bounds
Institute for Data Science in Mechanical Engineering, RWTH Aachen University（机械工程数据科学研究所，亚琛工业大学）；Department of Mathematics, School of Computation, Information and Technology, Technical University of Munich（数学系，计算、信息与技术学院，慕尼黑技术大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）
Kernel-Based Evaluation of Conditional Biological Sequence Models
Systems Biology, Harvard Medical School, Boston, USA（系统生物学，哈佛医学院，美国波士顿）；Harvard Medical School, Broad Institute, Boston, USA（哈佛医学院，Broad研究所，美国波士顿）；Department of Statistics, University of Oxford, Oxford, UK（统计学系，牛津大学，英国牛津）；Courant Institute, New York University, New York, USA（Courant研究所，纽约大学，美国纽约）
Practical Kernel Tests of Conditional Independence
McGill University and Mila（麦吉尔大学和Mila）；Gatsby Computational Neuroscience Unit, University College London Centre for Artificial Intelligence, University College London（Gatsby计算神经科学单元，伦敦大学学院人工智能中心，伦敦大学学院）；Microsoft AI（微软AI）；University of British Columbia and Amii（不列颠哥伦比亚大学和Amii）；Gatsby Computational Neuroscience Unit, University College London Google DeepMind（Gatsby计算神经科学单元，伦敦大学学院谷歌DeepMind）
Sequential Kernel-based Conditional Independence Testing via Adaptive Betting
Department of Computer Science, University of British Columbia, Vancouver, Canada（不列颠哥伦比亚大学计算机科学系，温哥华，加拿大）；Alberta Machine Intelligence Institute, Edmonton, Canada（阿尔伯塔机器智能研究所，爱德蒙顿，加拿大）
Google Research（谷歌研究院）；MPI for Intelligent Systems（马克斯·普朗克智能系统研究所）；University College London（伦敦大学学院）
We propose two nonparametric statistical tests of goodness of fit for conditional distributions: given a conditional probability density function $p(y|x)$ and a joint sample, decide whether the sample is drawn from $p(y|x)r_x(x)$ for some density $r_x$. Our tests, formulated with a Stein operator, can be applied to any differentiable conditional density model, and require no knowledge of the normalizing constant. We show that 1) our tests are consistent against any fixed alternative conditional model; 2) the statistics can be estimated easily, requiring no density estimation as an intermediate…
Conditional expectiles are becoming an increasingly important tool in finance as well as in other areas of applications. We analyse a support vector machine type approach for estimating conditional expectiles and establish learning rates that are minimax optimal modulo a logarithmic factor if Gaussian RBF kernels are used and the desired expectile is smooth in a Besov sense. As a special case, our learning rates improve the best known rates for kernel-based least squares regression in this scenario. Key ingredients of our statistical analysis are a general calibration inequality for the asymme…
Conditional KRR: Injecting Unpenalized Features into Kernel Methods with Applications to Kernel Thresholding
Department of Mathematics, Nazarbayev University, Astana, Kazakhstan（纳扎尔巴耶夫大学数学系）；Nazarbayev University Research Administration, Astana, Kazakhstan（纳扎尔巴耶夫大学研究行政部）；Purdue University Fort Wayne, Indiana, USA（普渡大学枫林分校）
As a vital problem in classification-oriented transfer, unsupervised domain adaptation (UDA) has attracted widespread attention in recent years. Previous UDA methods assume the marginal distributions of different domains are shifted while ignoring the discriminant information in the label distributions. This leads to classification performance degeneration in real applications. In this work, we focus on the conditional distribution shift problem which is of great concern to current conditional invariant models. We aim to seek a kernel covariance embedding for conditional distribution which rem…
Fast and Scalable Score-Based Kernel Calibration Tests
Gatsby Computational Neuroscience Unit（加布特计算神经科学单位）；University College London（伦敦大学学院）；Department of Information Technology（信息技术系）；Uppsala University（乌普萨拉大学）；Division of Statistics and Machine Learning（统计与机器学习 division）；Linköping University（林雪平大学）
