Practitioners of Bayesian statistics have long depended on Markov chain Monte Carlo (MCMC) to obtain samples from intractable posterior distributions. Unfortunately, MCMC algorithms are typically serial, and do not scale to the large datasets typical of modern machine learning. The recently proposed consensus Monte Carlo algorithm removes this limitation by partitioning the data and drawing samples conditional on each partition in parallel (Scott et al, 2013). A fixed aggregation function then combines these samples, yielding approximate posterior samples. We introduce variational consensus Monte Carlo (VCMC), a variational Bayes algorithm that optimizes over aggregation functions to obtain samples from a distribution that better approximates the target. The resulting objective contains an intractable entropy term; we therefore derive a relaxation of the objective and show that the relaxed problem is blockwise concave under mild conditions. We illustrate the advantages of our algorithm on three inference tasks from the literature, demonstrating both the superior quality of the posterior approximation and the moderate overhead of the optimization step. Our algorithm achieves a relative error reduction (measured against serial MCMC) of up to 39% compared to consensus Monte Carlo on the task of estimating 300-dimensional probit regression parameter expectations; similarly, it achieves an error reduction of 92% on the task of estimating cluster comembership probabilities in a Gaussian mixture model with 8 components in 8 dimensions. Furthermore, these gains come at moderate cost compared to the runtime of serial MCMC, achieving near-ideal speedup in some instances.
MetFlow: A New Efficient Method for Bridging the Gap between Markov Chain Monte Carlo and Variational Inference
CMAP, Ecole Polytechnique, Universite Paris-Saclay（巴黎萨克雷大学综合理工学院CMAP实验室）；CDISE, Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究所CDISE实验室）；Ecole Normale Supérieure Paris-Saclay（巴黎萨克雷高等师范学院）
University of Torontoland（多伦多兰大学）；University of Edenborrow（伊甸伯勒大学）；HSE University（高等经济学院）；Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）；University of Oxford（牛津大学）；Centre de recherche Lagrange en mathematiques et calcul（拉格朗日数学与计算研究中心）
Efficient Online Variational Estimation via Monte Carlo Sampling
Télécom Paris, Institut Polytechnique de Paris（巴黎电信学院，巴黎理工学院）；Université Bretagne Sud, UMR CNRS 6205, LMBA（南布列塔尼大学，法国国家科学研究中心联合研究单位6205，数学与建模实验室）；Sorbonne Université, Université Paris Cité, CNRS, Laboratoire de Probabilités, Statistique et Modélisation, LPSM（索邦大学，巴黎西岱大学，法国国家科学研究中心，概率、统计与建模实验室）
Variational Langevin Hamiltonian Monte Carlo for Distant Multi-modal Sampling
East China Normal University（华东师范大学）
Hamilton Monte Carlo（HMC）采样算法利用Hamilton动力学构造高效的Markov Chain Monte Carlo（MCMC），在机器学习和统计学中越来越受欢迎。由于HMC使用目标分布的梯度信息，它可以比随机行走建议更有效地探索状态空间。然而，涉及多模态分布的概率推理是非常困难的标准HMC方法，特别是当模式彼此远离。采样算法通常不能穿越低概率的地方。在本文中，我们提出了一种新的MCMC算法，旨在有效地从多峰分布采样。该方法改进了哈密顿动力学，以减少样本的自相关性，并使用变分分布来探索相空间，并找到新的模式。给出了一个形式化的证明，证明了该方法能够收敛到目标分布。合成和真实的数据集被用来评估其属性和性能。实验结果验证了理论的正确性，并显示出优越的性能，在多模态采样。
