arXiv 2103.010302021-03cs.LGstat.ML

An Easy to Interpret Diagnostic for Approximate Inference: Symmetric Divergence Over Simulations

Justin Domke

英文摘要：

It is important to estimate the errors of probabilistic inference algorithms. Existing diagnostics for Markov chain Monte Carlo methods assume inference is asymptotically exact, and are not appropriate for approximate methods like variational inference or Laplace's method. This paper introduces a diagnostic based on repeatedly simulating datasets from the prior and performing inference on each. The central observation is that it is possible to estimate a symmetric KL-divergence defined over these simulations.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Alpha-Beta Divergence For Variational Inference

University College of London（伦敦大学学院）

This paper introduces a variational approximation framework using direct optimization of what is known as the {\it scale invariant Alpha-Beta divergence} (sAB divergence). This new objective encompasses most variational objectives that use the Kullback-Leibler, the R{é}nyi or the gamma divergences. It also gives access to objective functions never exploited before in the context of variational inference. This is achieved via two easy to interpret control parameters, which allow for a smooth interpolation over the divergence space while trading-off properties such as mass-covering of a target d…

2018-05-20cs.LGstat.ML

Rethinking Collapsed Variational Bayes Inference for LDA

The University of Tokyo（东京大学）

We propose a novel interpretation of the collapsed variational Bayes inference with a zero-order Taylor expansion approximation, called CVB0 inference, for latent Dirichlet allocation (LDA). We clarify the properties of the CVB0 inference by using the alpha-divergence. We show that the CVB0 inference is composed of two different divergence projections: alpha=1 and -1. This interpretation will help shed light on CVB0 works.

2012-06-27cs.LGstat.ML

Constraining Variational Inference with Geometric Jensen-Shannon Divergence

University of Cambridge（剑桥大学）

We examine the problem of controlling divergences for latent space regularisation in variational autoencoders. Specifically, when aiming to reconstruct example $x\in\mathbb{R}^{m}$ via latent space $z\in\mathbb{R}^{n}$ ($n\leq m$), while balancing this against the need for generalisable latent representations. We present a regularisation mechanism based on the skew-geometric Jensen-Shannon divergence $\left(\textrm{JS}^{\textrm{G}_α}\right)$. We find a variation in $\textrm{JS}^{\textrm{G}_α}$, motivated by limiting cases, which leads to an intuitive interpolation between forward and reverse K…

2021-01-04cs.LGstat.ML

可解释性集成树的重建质量评估的一类发散度度量

A Family of Divergence Measures for Evaluating the Reconstruction Quality of Explainable Ensemble Trees

Department of Economics and Statistics, University of Naples Federico II（那不勒斯费德里科二世大学经济与统计系）

本文提出了一种基于发散度的度量框架，用于评估可解释性集成树的重建质量，通过区分一致性和关联性，提供了一种新的诊断方法来识别重建失败的具体原因。

2026-05-19cs.LGstat.ME

Easy Monotonic Policy Iteration

UC Berkeley（加州大学伯克利分校）

A key problem in reinforcement learning for control with general function approximators (such as deep neural networks and other nonlinear functions) is that, for many algorithms employed in practice, updates to the policy or $Q$-function may fail to improve performance---or worse, actually cause the policy performance to degrade. Prior work has addressed this for policy iteration by deriving tight policy improvement bounds; by optimizing the lower bound on policy improvement, a better policy is guaranteed. However, existing approaches suffer from bounds that are hard to optimize in practice be…

2016-02-29cs.AIcs.LGstat.ML

退火路径的变分表示：单调嵌入下的Bregman信息

Variational Representations of Annealing Paths: Bregman Information under Monotonic Embedding

Vector Institute（向量研究所）；USC Information Sciences Institute（南加州大学信息科学研究所）；Sony Computer Science Laboratories Inc（索尼计算机科学实验室公司）

本文通过单调嵌入下的Bregman散度框架，证明拟算术均值在变换后最小化期望散度，从而揭示退火路径中间密度的变分表示及其与常见散度泛函的关联。

2024-02-06cs.ITcs.LGmath.IT

更多相关论文收起更多论文

Model Fusion with Kullback--Leibler Divergence

MIT（麻省理工学院）；IBM Research（IBM研究院）

We propose a method to fuse posterior distributions learned from heterogeneous datasets. Our algorithm relies on a mean field assumption for both the fused model and the individual dataset posteriors and proceeds using a simple assign-and-average approach. The components of the dataset posteriors are assigned to the proposed global model components by solving a regularized variant of the assignment problem. The global components are then updated based on these assignments by their mean under a KL divergence. For exponential family variational distributions, our formulation leads to an efficien…

2020-07-13cs.LGstat.ML

NLS: an accurate and yet easy-to-interpret regression method

University College Cork（科克大学学院）；Budapest University of Technology and Economics（布达佩斯技术与经济大学）；University of São Paulo（圣保罗大学）；University of São Carlos（圣卡洛斯大学）

An important feature of successful supervised machine learning applications is to be able to explain the predictions given by the regression or classification model being used. However, most state-of-the-art models that have good predictive power lead to predictions that are hard to interpret. Thus, several model-agnostic interpreters have been developed recently as a way of explaining black-box classifiers. In practice, using these methods is a slow process because a novel fitting is required for each new testing instance, and several non-trivial choices must be made. We develop NLS (neural l…

2019-10-11cs.LGstat.MEstat.ML

ML可解释性：简单并不容易

ML Interpretability: Simple Isn't Easy

University of Bern（伯尔尼大学）

本文聚焦“可解释性光谱”的另一端，探讨线性模型、决策树及MARS和GAM等模型的可解释性本质，指出尽管获取可解释性的方式存在差异，但具体情形下的可解释性仍可被清晰阐释。

2022-11-24cs.CYcs.LG

Generalised Mixability, Constant Regret, and Bayesian Updating

The Australian National University（澳大利亚国立大学）；NICTA（澳大利亚国家信息与通信技术研究院）；Microsoft Research（微软研究院）

Mixability of a loss is known to characterise when constant regret bounds are achievable in games of prediction with expert advice through the use of Vovk's aggregating algorithm. We provide a new interpretation of mixability via convex analysis that highlights the role of the Kullback-Leibler divergence in its definition. This naturally generalises to what we call $Φ$-mixability where the Bregman divergence $D_Φ$ replaces the KL divergence. We prove that losses that are $Φ$-mixable also enjoy constant regret bounds via a generalised aggregating algorithm that is similar to mirror descent.

2014-03-10cs.LGstat.ML

语言模型间Kullback--Leibler散度的更好估计

Better Estimation of the Kullback--Leibler Divergence Between Language Models

ETH Zürich（苏黎世联邦理工学院）

本文提出一种无偏且方差更低的Rao-Blackwellized估计器来估计语言模型间的KL散度，并在情感控制微调中验证其稳定性，同时推导了梯度估计器以提升训练效果。

2025-10-25cs.AIcs.CLcs.LG

Explainer Divergence Scores (EDS): Some Post-Hoc Explanations May be Effective for Detecting Unknown Spurious Correlations

University of Toronto（多伦多大学）；University of Cambridge（剑桥大学）

Recent work has suggested post-hoc explainers might be ineffective for detecting spurious correlations in Deep Neural Networks (DNNs). However, we show there are serious weaknesses with the existing evaluation frameworks for this setting. Previously proposed metrics are extremely difficult to interpret and are not directly comparable between explainer methods. To alleviate these constraints, we propose a new evaluation methodology, Explainer Divergence Scores (EDS), grounded in an information theory approach to evaluate explainers. EDS is easy to interpret and naturally comparable across expla…

2022-11-14cs.AIcs.LG

Clarifying MCMC-based training of modern EBMs : Contrastive Divergence versus Maximum Likelihood

Mila（米拉人工智能研究所）；Université de Montréal（蒙特利尔大学）

The Energy-Based Model (EBM) framework is a very general approach to generative modeling that tries to learn and exploit probability distributions only defined though unnormalized scores. It has risen in popularity recently thanks to the impressive results obtained in image generation by parameterizing the distribution with Convolutional Neural Networks (CNN). However, the motivation and theoretical foundations behind modern EBMs are often absent from recent papers and this sometimes results in some confusion. In particular, the theoretical justifications behind the popular MCMC-based learning…

2022-02-24cs.LG

Sinkhorn Divergence of Topological Signature Estimates for Time Series Classification

School of Computing, Electronics and Mathematics, Coventry University UK（考文垂大学计算机、电子与数学系）

Distinguishing between classes of time series sampled from dynamic systems is a common challenge in systems and control engineering, for example in the context of health monitoring, fault detection, and quality control. The challenge is increased when no underlying model of a system is known, measurement noise is present, and long signals need to be interpreted. In this paper we address these issues with a new non parametric classifier based on topological signatures. Our model learns classes as weighted kernel density estimates (KDEs) over persistent homology diagrams and predicts new traject…

2019-02-14cs.LGnlin.CDstat.ML