Gaussian processes retain the linear model either as a special case, or in the limit. We show how this relationship can be exploited when the data are at least partially linear. However from the perspective of the Bayesian posterior, the Gaussian processes which encode the linear model either have probability of nearly zero or are otherwise unattainable without the explicit construction of a prior with the limiting linear model in mind. We develop such a prior, and show that its practical benefits extend well beyond the computational and conceptual simplicity of the linear model. For example, linearity can be extracted on a per-dimension basis, or can be combined with treed partition models to yield a highly efficient nonstationary model. Our approach is demonstrated on synthetic and real datasets of varying linearity and dimensionality.
Gaussian process priors are commonly used in aerospace design for performing Bayesian optimization. Nonetheless, Gaussian processes suffer two significant drawbacks: outliers are a priori assumed unlikely, and the posterior variance conditioned on observed data depends only on the locations of those data, not the associated sample values. Student's-T processes are a generalization of Gaussian processes, founded on the Student's-T distribution instead of the Gaussian distribution. Student's-T processes maintain the primary advantages of Gaussian processes (kernel function, analytic update rule)…
We propose a simple method that combines neural networks and Gaussian processes. The proposed method can estimate the uncertainty of outputs and flexibly adjust target functions where training data exist, which are advantages of Gaussian processes. The proposed method can also achieve high generalization performance for unseen input configurations, which is an advantage of neural networks. With the proposed method, neural networks are used for the mean functions of Gaussian processes. We present a scalable stochastic inference procedure, where sparse Gaussian processes are inferred by stochast…
Expected-posterior priors (EPP) have been proved to be extremely useful for testing hypothesis on the regression coefficients of normal linear models. One of the advantages of using EPPs is that impropriety of baseline priors causes no indeterminacy. However, in regression problems, they based on one or more \textit{training samples}, that could influence the resulting posterior distribution. The power-expected-posterior priors are minimally-informative priors that diminishing the effect of training samples on the EPP approach, by combining ideas from the power-prior and unit-information-prior…
We investigate the Student-t process as an alternative to the Gaussian process as a nonparametric prior over functions. We derive closed form expressions for the marginal likelihood and predictive distribution of a Student-t process, by integrating away an inverse Wishart process prior over the covariance kernel of a Gaussian process model. We show surprising equivalences between different hierarchical Gaussian process models leading to Student-t processes, and derive a new sampling scheme for the inverse Wishart process, which helps elucidate these equivalences. Overall, we show that a Studen…
Aalto university（阿尔托大学）；Helsinki Institute for Information Technology HIIT（赫尔辛基信息技术研究所（HIIT））
We propose deep convolutional Gaussian processes, a deep Gaussian process architecture with convolutional structure. The model is a principled Bayesian framework for detecting hierarchical combinations of local features for image classification. We demonstrate greatly improved image classification performance compared to current Gaussian process approaches on the MNIST and CIFAR-10 datasets. In particular, we improve CIFAR-10 accuracy by over 10 percentage points.
Bayesian learning using Gaussian processes provides a foundational framework for making decisions in a manner that balances what is known with what could be learned by gathering data. In this dissertation, we develop techniques for broadening the applicability of Gaussian processes. This is done in two ways. Firstly, we develop pathwise conditioning techniques for Gaussian processes, which allow one to express posterior random functions as prior random functions plus a dependent update term. We introduce a wide class of efficient approximations built from this viewpoint, which can be randomly …
Gaussian processes are used in machine learning to learn input-output mappings from observed data. Gaussian process regression is based on imposing a Gaussian process prior on the unknown regressor function and statistically conditioning it on the observed data. In system identification, Gaussian processes are used to form time series prediction models such as non-linear finite-impulse response (NFIR) models as well as non-linear autoregressive (NARX) models. Gaussian process state-space models (GPSS) can be used to learn the dynamic and measurement models for a state-space representation of t…
University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）
Gaussian processes are one of the dominant approaches in Bayesian learning. Although the approach has been applied to numerous problems with great success, it has a few fundamental limitations. Multiple methods in literature have addressed these limitations. However, there has not been a comprehensive survey of the topics as of yet. Most existing surveys focus on only one particular variant of Gaussian processes and their derivatives. This survey details the core motivations for using Gaussian processes, their mathematical formulations, limitations, and research themes that have flourished ove…
Université de Lille（里尔大学）；CNRS/CRIStAL（法国国家科学研究中心/CRIStAL实验室）；Inria Lille（法国里尔国家信息与自动化研究所）
Bayesian Optimization using Gaussian Processes is a popular approach to deal with the optimization of expensive black-box functions. However, because of the a priori on the stationarity of the covariance matrix of classic Gaussian Processes, this method may not be adapted for non-stationary functions involved in the optimization problem. To overcome this issue, a new Bayesian Optimization approach is proposed. It is based on Deep Gaussian Processes as surrogate models instead of classic Gaussian Processes. This modeling technique increases the power of representation to capture the non-station…
As a generalization of the work in [Lee et al., 2017], this note briefly discusses when the prior of a neural network output follows a Gaussian process, and how a neural-network-induced Gaussian process is formulated. The posterior mean functions of such a Gaussian process regression lie in the reproducing kernel Hilbert space defined by the neural-network-induced kernel. In the case of two-layer neural networks, the induced Gaussian processes provide an interpretation of the reproducing kernel Hilbert spaces whose union forms a Barron space.
Gaussian processes are ubiquitous in nature and engineering. A case in point is a class of neural networks in the infinite-width limit, whose priors correspond to Gaussian processes. Here we perturbatively extend this correspondence to finite-width neural networks, yielding non-Gaussian processes as priors. The methodology developed herein allows us to track the flow of preactivation distributions by progressively integrating out random variables from lower to higher layers, reminiscent of renormalization-group flow. We further develop a perturbative procedure to perform Bayesian inference wit…
We provide the strong approximation of empirical copula processes by a Gaussian process. In addition we establish a strong approximation of the smoothed empirical copula processes and a law of iterated logarithm.
Various approaches to stochastic processes exist, noting that key properties such as measurability and continuity are not trivially satisfied. We introduce a new theory for Gaussian processes using improper linear functionals. Using a collection of i.i.d. standard normal variables, we define Gaussian white noise and discuss its properties. This is extended to general Gaussian processes on Hilbert space, where the variance is allowed to be any suitable operator. Our main focus is $L^2$ spaces, and we discuss criteria for Gaussian processes to be continuous in this setting. Finally, we outline a…
Australian Centre for Field Robotics, The University of Sydney（悉尼大学澳大利亚现场机器人中心）
This paper evaluates heterogeneous information fusion using multi-task Gaussian processes in the context of geological resource modeling. Specifically, it empirically demonstrates that information integration across heterogeneous information sources leads to superior estimates of all the quantities being modeled, compared to modeling them individually. Multi-task Gaussian processes provide a powerful approach for simultaneous modeling of multiple quantities of interest while taking correlations between these quantities into consideration. Experiments are performed on large scale real sensor da…
We propose a novel Bayesian nonparametric method to learn translation-invariant relationships on non-Euclidean domains. The resulting graph convolutional Gaussian processes can be applied to problems in machine learning for which the input observations are functions with domains on general graphs. The structure of these models allows for high dimensional inputs while retaining expressibility, as is the case with convolutional neural networks. We present applications of graph convolutional Gaussian processes to images and triangular meshes, demonstrating their versatility and effectiveness, com…
A gentle introduction to Gaussian processes (GPs). The three parts of the document consider GPs for regression, classification, and dimensionality reduction.
Real-world measurement noise in applications like robotics is often correlated in time, but we typically assume i.i.d. Gaussian noise for filtering. We propose general Gaussian Processes as a non-parametric model for correlated measurement noise that is flexible enough to accurately reflect correlation in time, yet simple enough to enable efficient computation. We show that this model accurately reflects the measurement noise resulting from vision-based Simultaneous Localization and Mapping (SLAM), and argue that it provides a flexible means of modeling measurement noise for a wide variety of …
In this paper, we study central and non-central limit theorems for partial sum of functionals of general stationary Gaussian fields. We apply our result to study drift parameter estimation problems for some stochastic differential equations related to stationary Gaussian processes.
The asymptotic analysis of covariance parameter estimation of Gaussian processes has been subject to intensive investigation. However, this asymptotic analysis is very scarce for non-Gaussian processes. In this paper, we study a class of non-Gaussian processes obtained by regular non-linear transformations of Gaussian processes. We provide the increasing-domain asymptotic properties of the (Gaussian) maximum likelihood and cross validation estimators of the covariance parameters of a non-Gaussian process of this class. We show that these estimators are consistent and asymptotically normal, alt…
Deep Learning Gaussian Processes (DL-GP) are proposed as a methodology for analyzing (approximating) computer models that produce heteroskedastic and high-dimensional output. Computer simulation models have many areas of applications, including social-economic processes, agriculture, environmental, biology, engineering and physics problems. A deterministic transformation of inputs is performed by deep learning and predictions are calculated by traditional Gaussian Processes. We illustrate our methodology using a simulation of motorcycle accidents and simulations of an Ebola outbreak. Finally, …
There are all kinds of weak dependence. For example, strong mixing. Short-range dependence (SRD) is also a form of weak dependence. It occurs in the context of processes that are subordinated to the Gaussian. Is a SRD process strong mixing if the underlying Gaussian process is long-range dependent? We show that this is not necessarily the case.
In this paper we derive Piterbarg's max-discretisation theorem for two different grids considering centered stationary vector Gaussian processes. So far in the literature results in this direction have been derived for the joint distribution of the maximum of Gaussian processes over $[0,T]$ and over a grid $ \mathfrak{R}(δ_1(T))=\{kδ_1(T): k=0,1,\cdots\}$. In this paper we extend recent findings by considering additionally the \bE{maximum} over another grid $ \mathfrak{R}(δ_2(T))$. We derive the joint limiting distribution of maximum of stationary Gaussian vector processes for different choice…
We present a non-parametric joint estimation method for fMRI task activation values and the hemodynamic response function (HRF). The HRF is modeled as a Gaussian process, making continuous evaluation possible for jittered paradigms and providing a variance estimate at each point.
Gaussian processes occupy one of the leading places in modern statistics and probability theory due to their importance and a wealth of strong results. The common use of Gaussian processes is in connection with problems related to estimation, detection, and many statistical or machine learning models. With the fast development of Gaussian process applications, it is necessary to consolidate the fundamentals of vector-valued stochastic processes, in particular multivariate Gaussian processes, which is the essential theory for many applied problems with multiple correlated responses. In this pap…
In this paper, we study the quantitative behavior of a spatial risk measure corresponding to a damage function and a region, taking into account the spatial dependence of the underlying process. This kind of risk measure has already been introduced and studied for some max-stable processes in [Koch2015]. In this paper, we consider isotropic Gaussian processes and the excess damage function over a threshold. We performed a simulation study and a real data study.
We introduce a class of spatial stochastic processes in the max-domain of attraction of familiar max-stable processes. The new class is based on Cox processes and comprises models with short range dependence. We show that statistical inference is possible within the given framework, at least under some reasonable restrictions.
