Sampling from unnormalized densities using diffusion models has emerged as a powerful paradigm. However, while recent approaches that use least-squares `matching' objectives have improved scalability, they often necessitate significant trade-offs, such as restricting prior distributions or relying on unstable optimization schemes. By generalizing these methods as special forms of fixed-point iterations rooted in Nelson's relation, we develop a new method that addresses these limitations, called Bridge Matching Sampler (BMS). Our approach enables learning a stochastic transport map between arbitrary prior and target distributions with a single, scalable, and stable objective. Furthermore, we introduce a damped variant of this iteration that incorporates a regularization term to mitigate mode collapse and further stabilize training. Empirically, we demonstrate that our method enables sampling at unprecedented scales while preserving mode diversity, achieving state-of-the-art results on complex synthetic densities and high-dimensional molecular benchmarks.
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；Google Research（谷歌研究院）
We propose a general and scalable approximate sampling strategy for probabilistic models with discrete variables. Our approach uses gradients of the likelihood function with respect to its discrete inputs to propose updates in a Metropolis-Hastings sampler. We show empirically that this approach outperforms generic samplers in a number of difficult settings including Ising models, Potts models, restricted Boltzmann machines, and factorial hidden Markov models. We also demonstrate the use of our improved sampler for training deep energy-based models on high dimensional discrete data. This appro…
Entropy Across the Bridge: Conditional-Marginal Discretization for Flow and Schrödinger Samplers
NVIDIA Corporation（NVIDIA公司）；University of Oxford（牛津大学）；Donders Institute for Brain, Cognition, and Behaviour（大脑与行为研究所）；AITHYRA, Research Institute for Biomedical AI（生物医学人工智能研究所）
We propose discrete Langevin proposal (DLP), a simple and scalable gradient-based proposal for sampling complex high-dimensional discrete distributions. In contrast to Gibbs sampling-based methods, DLP is able to update all coordinates in parallel in a single step and the magnitude of changes is controlled by a stepsize. This allows a cheap and efficient exploration in the space of high-dimensional and strongly correlated variables. We prove the efficiency of DLP by showing that the asymptotic bias of its stationary distribution is zero for log-quadratic distributions, and is small for distrib…
Population synthesis is concerned with the generation of synthetic yet realistic representations of populations. It is a fundamental problem in the modeling of transport where the synthetic populations of micro-agents represent a key input to most agent-based models. In this paper, a new methodological framework for how to 'grow' pools of micro-agents is presented. The model framework adopts a deep generative modeling approach from machine learning based on a Variational Autoencoder (VAE). Compared to the previous population synthesis approaches, including Iterative Proportional Fitting (IPF),…
Deep unfolding of MCMC kernels: scalable, modular & explainable GANs for high-dimensional posterior sampling
Maxwell Institute for Mathematical Sciences（麦克斯韦数学科学研究所）；School of Mathematical and Computer Sciences, Heriot-Watt University（赫瑞-瓦特大学数学与计算机科学学院）；School of Mathematics, University of Edinburgh（爱丁堡大学数学学院）
University College of London（伦敦大学学院）；Athens University of Economics and Business（雅典经济与商业大学）；The Alan Turing Institute（艾伦·图灵研究所）
We present a scalable approach to performing approximate fully Bayesian inference in generic state space models. The proposed method is an alternative to particle MCMC that provides fully Bayesian inference of both the dynamic latent states and the static parameters of the model. We build up on recent advances in computational statistics that combine variational methods with sequential Monte Carlo sampling and we demonstrate the advantages of performing full Bayesian inference over the static parameters rather than just performing variational EM approximations. We illustrate how our approach e…
The Schrödinger bridge problem (SBP) finds the most likely stochastic evolution between two probability distributions given a prior stochastic evolution. As well as applications in the natural sciences, problems of this kind have important applications in machine learning such as dataset alignment and hypothesis testing. Whilst the theory behind this problem is relatively mature, scalable numerical recipes to estimate the Schrödinger bridge remain an active area of research. We prove an equivalence between the SBP and maximum likelihood estimation enabling direct application of successful mach…
Dept. of Comp. Sci. & Tech., Institute for AI, BNRist Center（计算机科学与技术系，人工智能研究所，BNRist中心）；Tsinghua-Bosch Joint ML Center, Tsinghua University（清华大学-博世联合机器学习中心，清华大学）；Shengshu Technology（盛舒科技）；Pazhou Lab (Huangpu), Guangzhou, China（黄埔实验室（广州），中国）
Technical University Munich, Chair of Robotics, Artificial Intelligence and Embedded Systems（慕尼黑技术大学机器人、人工智能与嵌入式系统教研室）；ELLIS Unit Linz, LIT AI Lab, Johannes Kepler University Linz, Austria（林茨ELLIS单位，LIT人工智能实验室，林茨约翰内斯·开普勒大学，奥地利）；NXAI Lab & NXAI GmbH, Linz, Austria（NXAI实验室及NXAI GmbH，林茨，奥地利）
Harvard Data Science Initiative（哈佛数据科学倡议）；Harvard University（哈佛大学）
Uncertainty quantification has been a core of the statistical machine learning, but its computational bottleneck has been a serious challenge for both Bayesians and frequentists. We propose a model-based framework in quantifying uncertainty, called predictive-matching Generative Parameter Sampler (GPS). This procedure considers an Uncertainty Quantification (UQ) distribution on the targeted parameter, which matches the corresponding predictive distribution to the observed data. This framework adopts a hierarchical modeling perspective such that each observation is modeled by an individual para…
We present Path Integral Sampler~(PIS), a novel algorithm to draw samples from unnormalized probability density functions. The PIS is built on the Schrödinger bridge problem which aims to recover the most likely evolution of a diffusion process given its initial distribution and terminal distribution. The PIS draws samples from the initial distribution and then propagates the samples through the Schrödinger bridge to reach the terminal distribution. Applying the Girsanov theorem, with a simple prior diffusion, we formulate the PIS as a stochastic optimal control problem whose running cost is t…
Scalable Logistic Gaussian Process Density Regression with Kinetic Langevin Sampling
Nanyang Technological University（南洋理工大学）；Eötvös Loránd University (ELTE)（厄特沃什·罗兰大学（ELTE））；HUN-REN Institute for Computer Science and Control (SZTAKI)（匈牙利国家研究网络计算机科学与控制研究所（SZTAKI））
Beijing University of Posts and Telecommunications（北京邮电大学）；Peking University（北京大学）；Hong Kong University of Science and Technology（香港科技大学）
Network representation learning (NRL) technique has been successfully adopted in various data mining and machine learning applications. Random walk based NRL is one popular paradigm, which uses a set of random walks to capture the network structural information, and then employs word2vec models to learn the low-dimensional representations. However, until now there is lack of a framework, which unifies existing random walk based NRL models and supports to efficiently learn from large networks. The main obstacle comes from the diverse random walk models and the inefficient sampling method for th…
Harvard University（哈佛大学）；Microsoft Research New England（微软新英格兰研究院）
Spike-and-slab priors are commonly used for Bayesian variable selection, due to their interpretability and favorable statistical properties. However, existing samplers for spike-and-slab posteriors incur prohibitive computational costs when the number of variables is large. In this article, we propose Scalable Spike-and-Slab ($S^3$), a scalable Gibbs sampling implementation for high-dimensional Bayesian regression with the continuous spike-and-slab prior of George and McCulloch (1993). For a dataset with $n$ observations and $p$ covariates, $S^3$ has order $\max\{ n^2 p_t, np \}$ computational…
University of Southern California（南加利福尼亚大学）；US Army Research Lab（美国陆军研究实验室）
Graph Neural Networks (GNNs) are powerful deep learning models to generate node embeddings on graphs. When applying deep GNNs on large graphs, it is still challenging to perform training in an efficient and scalable way. We propose a novel parallel training framework. Through sampling small subgraphs as minibatches, we reduce training workload by orders of magnitude compared with state-of-the-art minibatch methods. We then parallelize the key computation steps on tightly-coupled shared memory systems. For graph sampling, we exploit parallelism within and across sampler instances, and propose a…
Scalable Discrete Diffusion Samplers: Combinatorial Optimization and Statistical Physics
ELLIS Unit Linz, LIT AI Lab, Johannes Kepler University Linz, Austria（林茨ELLIS单元、LIT AI实验室、林茨约瑟夫·冯·海德里希大学，奥地利）；Department of Electrical & Computer Engineering, Georgia Institute of Technology（电气与计算机工程系，佐治亚理工学院）；NXAI Lab & NXAI GmbH, Linz, Austria（NXAI实验室及NXAI GmbH，林茨，奥地利）
Georgia Institute of Technology（佐治亚理工学院）；Amazon Shanghai AI Lab（亚马逊上海人工智能实验室）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Graph neural networks (GNN) have recently emerged as a vehicle for applying deep network architectures to graph and relational data. However, given the increasing size of industrial datasets, in many practical situations the message passing computations required for sharing information across GNN layers are no longer scalable. Although various sampling methods have been introduced to approximate full-graph training within a tractable budget, there remain unresolved complications such as high variances and limited theoretical guarantees. To address these issues, we build upon existing work and …
Microcanonical Langevin Ensembles: Advancing the Sampling of Bayesian Neural Networks
LMU Munich（慕尼黑大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；University of California Berkeley（加州大学伯克利分校）；Eraneos Analytics Germany GmbH（埃拉诺斯分析德国有限公司）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）
BFMT: Enhancing Search Capabilities of Tree Sampler via Bootstrap Flow-Map Tree
Department of CSE, Washington University in St.Louis, USA（美国圣路易斯华盛顿大学计算机科学与工程系）；Department of Information Technology, Uppsala University, Sweden（瑞典乌普萨拉大学信息技术系）
University of Bologna（博洛尼亚大学）；University of Trento（特伦托大学）；ETH Zurich（苏黎世联邦理工学院）
Modern real-time Structural Health Monitoring systems can generate a considerable amount of information that must be processed and evaluated for detecting early anomalies and generating prompt warnings and alarms about the civil infrastructure conditions. The current cloud-based solutions cannot scale if the raw data has to be collected from thousands of buildings. This paper presents a full-stack deployment of an efficient and scalable anomaly detection pipeline for SHM systems which does not require sending raw data to the cloud but relies on edge computation. First, we benchmark three algor…
Towards Intrinsically Calibrated Uncertainty Quantification in Industrial Data-Driven Models via Diffusion Sampler
College of Control Science and Engineering, Zhejiang University（浙江大学控制科学与工程学院）；Electrical Engineering Department, Polytechnique Montreal（蒙特利尔理工学院电气工程系）；GERAD；State Key Lab of General AI, School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院通用人工智能国家重点实验室）；Guangdong Provincial Key Laboratory of Petrochemical Equipment Fault Diagnosis, Guangdong University of Petrochemical Technology（广东石油化工学院广东省石化装备故障诊断重点实验室）
Yale University（耶鲁大学）；Criteo AI Lab（Criteo人工智能实验室）；Google Research（谷歌研究院）；Facebook AI Research（脸书人工智能研究院）
A determinantal point process (DPP) on a collection of $M$ items is a model, parameterized by a symmetric kernel matrix, that assigns a probability to every subset of those items. Recent work shows that removing the kernel symmetry constraint, yielding nonsymmetric DPPs (NDPPs), can lead to significant predictive performance gains for machine learning applications. However, existing work leaves open the question of scalable NDPP sampling. There is only one known DPP sampling algorithm, based on Cholesky decomposition, that can directly apply to NDPPs as well. Unfortunately, its runtime is cubi…
Iterated Denoising Energy Matching for Sampling from Boltzmann Densities
Mila – Québec AI Institute（米拉-魁北克人工智能研究所）；McGill University（麦吉尔大学）；Dreamfold；Université de Montréal（蒙特利尔大学）；University of Oxford（牛津大学）；Ciela Institute（Ciela研究所）；Center for Computational Astrophysics, Flatiron Institute（弗拉铁隆研究所计算天体物理中心）；Jagiellonian University（雅盖隆大学）；CIFAR（加拿大高级研究所）
Learning a Bayesian network (BN) from data can be useful for decision-making or discovering causal relationships. However, traditional methods often fail in modern applications, which exhibit a larger number of observed variables than data points. The resulting uncertainty about the underlying network as well as the desire to incorporate prior information recommend a Bayesian approach to learning the BN, but the highly combinatorial structure of BNs poses a striking challenge for inference. The current state-of-the-art methods such as order MCMC are faster than previous methods but prevent the…
Deep Reinforcement Learning (DRL) underlies in a simulated environment and optimizes objective goals. By extending the conventional interaction scheme, this paper proffers gym-ds3, a scalable and reproducible open environment tailored for a high-fidelity Domain-Specific System-on-Chip (DSSoC) application. The simulation corroborates to schedule hierarchical jobs onto heterogeneous System-on-Chip (SoC) processors and bridges the system to reinforcement learning research. We systematically analyze the representative SoC simulator and discuss the primary challenging aspects that the system (1) co…
