Probabilistic ordinary differential equation (ODE) solvers have been introduced over the past decade as uncertainty-aware numerical integrators. They typically proceed by assuming a functional prior to the ODE solution, which is then queried on a grid to form a posterior distribution over the ODE solution. As the queries span the integration interval, the approximate posterior solution then converges to the true deterministic one. Gaussian ODE filters, in particular, have enjoyed a lot of attention due to their computational efficiency, the simplicity of their implementation, as well as their provable fast convergence rates. In this article, we extend the methodology to stochastic differential equations (SDEs) and propose a probabilistic simulator for SDEs. Our approach involves transforming the SDE into a sequence of random ODEs using piecewise differentiable approximations of the Brownian motion. We then apply probabilistic ODE solvers to the individual ODEs, resulting in a pathwise probabilistic solution to the SDE\@. We establish worst-case strong $1.5$ local and $1.0$ global convergence orders for a specific instance of our method. We further show how we can marginalise the Brownian approximations, by incorporating its coefficients as part of the prior ODE model, allowing for computing exact transition densities under our model. Finally, we numerically validate the theoretical findings, showcasing reasonable weak convergence properties in the marginalised version.
Markov chain Monte Carlo (MCMC) samplers are numerical methods for drawing samples from a given target probability distribution. We discuss one particular MCMC sampler, the MALA-within-Gibbs sampler, from the theoretical and practical perspectives. We first show that the acceptance ratio and step size of this sampler are independent of the overall problem dimension when (i) the target distribution has sparse conditional structure, and (ii) this structure is reflected in the partial updating strategy of MALA-within-Gibbs. If, in addition, the target density is block-wise log-concave, then the s…
The logistic linear mixed model (LLMM) is one of the most widely used statistical models. Generally, Markov chain Monte Carlo algorithms are used to explore the posterior densities associated with the Bayesian LLMMs. Polson, Scott and Windle's (2013) Polya-Gamma data augmentation (DA) technique can be used to construct full Gibbs (FG) samplers for the LLMMs. Here, we develop efficient block Gibbs (BG) samplers for Bayesian LLMMs using the Polya-Gamma DA method. We compare the FG and BG samplers in the context of a real data example, as the correlation between the fixed effects and the random e…
Sampling from probability distributions is an important problem in statistics and machine learning, specially in Bayesian inference when integration with respect to posterior distribution is intractable and sampling from the posterior is the only viable option for inference. In this paper, we propose Schrödinger-Föllmer sampler (SFS), a novel approach for sampling from possibly unnormalized distributions. The proposed SFS is based on the Schrödinger-Föllmer diffusion process on the unit interval with a time dependent drift term, which transports the degenerate distribution at time zero to the …
We consider Gibbs and block Gibbs samplers for a Bayesian hierarchical version of the one-way random effects model. Drift and minorization conditions are established for the underlying Markov chains. The drift and minorization are used in conjunction with results from J. S. Rosenthal [J. Amer. Statist. Assoc. 90 (1995) 558-566] and G. O. Roberts and R. L. Tweedie [Stochastic Process. Appl. 80 (1999) 211-229] to construct analytical upper bounds on the distance to stationarity. These lead to upper bounds on the amount of burn-in that is required to get the chain within a prespecified (total var…
We revisit the theoretical properties of Hamiltonian stochastic differential equations (SDES) for Bayesian posterior sampling, and we study the two types of errors that arise from numerical SDE simulation: the discretization error and the error due to noisy gradient estimates in the context of data subsampling. Our main result is a novel analysis for the effect of mini-batches through the lens of differential operator splitting, revising previous literature results. The stochastic component of a Hamiltonian SDE is decoupled from the gradient noise, for which we make no normality assumptions. T…
In Bayesian inverse problems, one aims at characterizing the posterior distribution of a set of unknowns, given indirect measurements. For non-linear/non-Gaussian problems, analytic solutions are seldom available: Sequential Monte Carlo samplers offer a powerful tool for approximating complex posteriors, by constructing an auxiliary sequence of densities that smoothly reaches the posterior. Often the posterior depends on a scalar hyper-parameter. In this work, we show that properly designed Sequential Monte Carlo (SMC) samplers naturally provide an approximation of the marginal likelihood asso…
We consider Markov chain Monte Carlo (MCMC) algorithms for Bayesian high-dimensional regression with continuous shrinkage priors. A common challenge with these algorithms is the choice of the number of iterations to perform. This is critical when each iteration is expensive, as is the case when dealing with modern data sets, such as genome-wide association studies with thousands of rows and up to hundred of thousands of columns. We develop coupling techniques tailored to the setting of high-dimensional regression with shrinkage priors, which enable practical, non-asymptotic diagnostics of conv…
The emergence of big data has led to so-called convergence complexity analysis, which is the study of how Markov chain Monte Carlo (MCMC) algorithms behave as the sample size, $n$, and/or the number of parameters, $p$, in the underlying data set increase. This type of analysis is often quite challenging, in part because existing results for fixed $n$ and $p$ are simply not sharp enough to yield good asymptotic results. One of the first convergence complexity results for an MCMC algorithm on a continuous state space is due to Yang and Rosenthal (2019), who established a mixing time result for a…
The Gibbs sampler of Park and Casella is one of the most popular MCMC methods for sampling from the posterior density of the Bayesian Lasso regression. As with many Markov chain samplers, their Gibbs sampler lacks a theoretically sound method of output analysis --- a method for estimating the variance of a given ergodic average and estimating how closely the chain is sampling from the stationary distribution, that is, the burn-in. In this paper, we address this shortcoming by identifying regenerative structure in the sampler of Park and Casella, thus providing a theoretically sound method of a…
Bayesian inference with nested sampling requires a likelihood-restricted prior sampling method, which draws samples from the prior distribution that exceed a likelihood threshold. For high-dimensional problems, Markov Chain Monte Carlo derivatives have been proposed. We numerically study ten algorithms based on slice sampling, hit-and-run and differential evolution algorithms in ellipsoidal, non-ellipsoidal and non-convex problems from 2 to 100 dimensions. Mixing capabilities are evaluated with the nested sampling shrinkage test. This makes our results valid independent of how heavy-tailed the…
In computed tomography, data consist of measurements of the attenuation of X-rays passing through an object. The goal is to reconstruct the linear attenuation coefficient of the object's interior. For each position of the X-ray source, characterized by its angle with respect to a fixed coordinate system, one measures a set of data referred to as a view. A common assumption is that these view angles are known, but in some applications they are known with imprecision. We propose a framework to solve a Bayesian inverse problem that jointly estimates the view angles and an image of the object's at…
Pathwise convergence of a novel numerical scheme based on semi-implicit method for stochastic differential-algebraic equations with non-global Lipschitz coefficients
This paper is concerned with high moment and pathwise error estimates for fully discrete mixed finite element approximattions of stochastic Navier-Stokes equations with general additive noise. The implicit Euler-Maruyama scheme and standard mixed finite element methods are employed respectively for the time and space discretizations. High moment error estimates for both velocity and a time-avraged pressure approximations in strong $L^2$ and energy norms are obtained, pathwise error estimates are derived by using the Kolmogorov Theorem. Unlike their derterministic counterparts, the spatial erro…
Royal Holloway, University of London（伦敦大学皇家霍洛威学院）；National Physical Laboratory（英国国家物理实验室）；Université Lyon 2（里昂第二大学）；UCLouvain（瓦隆-布拉班特天主教大学）；University of Oxford（牛津大学）
Many inverse problems arising in applications come from continuum models where the unknown parameter is a field. In practice the unknown field is discretized resulting in a problem in $\mathbb{R}^N$, with an understanding that refining the discretization, that is increasing $N$, will often be desirable. In the context of Bayesian inversion this situation suggests the importance of two issues: (i) defining hyper-parameters in such a way that they are interpretable in the continuum limit $N \to \infty$ and so that their values may be compared between different discretization levels; (ii) underst…
This paper proposes a new pathwise sensitivity estimator for chaotic SDEs. By introducing a spring term between the original and perturbated SDEs, we derive a new estimator by importance sampling. The variance of the new estimator increases only linearly in time $T,$ compared with the exponential increase of the standard pathwise estimator. We compare our estimator with the Malliavin estimator and extend both of them to the Multilevel Monte Carlo method, which further improves the computational efficiency. Finally, we also consider using this estimator for the SDE with small volatility to appr…
The recent statistical finite element method (statFEM) provides a coherent statistical framework to synthesise finite element models with observed data. Through embedding uncertainty inside of the governing equations, finite element solutions are updated to give a posterior distribution which quantifies all sources of uncertainty associated with the model. However to incorporate all sources of uncertainty, one must integrate over the uncertainty associated with the model parameters, the known forward problem of uncertainty quantification. In this paper, we make use of Langevin dynamics to solv…
In this article, we consider numerical schemes for polynomial diffusions on the unit ball, which are solutions of stochastic differential equations with a diffusion coefficient of the form $\sqrt{1-|x|^{2}}$. We introduce a semi-implicit Euler--Maruyama scheme with the projection onto the unit ball and provide the $L^{2}$-rate of convergence. The main idea to consider the numerical scheme is the transformation argument introduced by Swart for proving the pathwise uniqueness for some stochastic differential equation with a non-Lipschitz diffusion coefficient.
Deep Truncated FBSDE Method for High-Dimensional Nonlinear PDEs and Fully Coupled FBSDEs
Research Institute of Intelligent Complex Systems, Fudan University（复旦大学智能复杂系统研究院）；Department of Mathematics, Friedrich-Alexander-Universität Erlangen–Nürnberg（埃尔朗根-纽伦堡大学数学系）
We investigate the stochastic modified equation which plays an important role in the stochastic backward error analysis for explaining the mathematical mechanism of a numerical method. The contribution of this paper is threefold. First, we construct a new type of stochastic modified equation, which is a perturbation of the Wong--Zakai approximation of the rough differential equation. For a symplectic method applied to a rough Hamiltonian system, the associated stochastic modified equation is proved to have a Hamiltonian formulation. Second, the pathwise convergence order of the truncated modif…
Weak Generative Sampler to Efficiently Sample Invariant Distribution of Stochastic Differential Equation
Department of Data Science, City University of Hong Kong（数据科学系，香港城市大学）；Institute of Natural Sciences and School of Mathematical Sciences, Shanghai Jiao Tong University（自然科学院和数学科学学院，上海交通大学）；Department of Mathematics and Department of Data Science, City University of Hong Kong（数学系和数据科学系，香港城市大学）
