解开固化、数据增强和先验在冷后验效应中的作用
Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effect
摘要:
贝叶斯深度学习中的“冷后验效应”(CPE)描述了一种令人不安的观察结果,即如果使用温度参数T<1人为锐化贝叶斯后验,贝叶斯神经网络的预测性能可以显著提高。CPE在理论和实践上都存在问题,自从这种效应被发现以来,许多研究人员提出了一些假设来解释这种现象。然而,尽管进行了大量的研究,但对这种影响仍然知之甚少。在这项工作中,我们提供了新的和微妙的证据相关的现有解释冷后效应,解开三个假设:1。Aitchison(2020)的数据集策展假设:我们根据经验表明,CPE不会出现在真实的策展数据集中,而是可以在具有不同策展强度的受控实验中产生。2. Izmailov等人(2021)和Fortuin等人(2021)的数据增强假设:我们根据经验证明,数据增强是足够的,但不是CPE存在的必要条件。3. Wenzel等人的坏先验假设。(2020):我们使用一个简单的实验来评估先验和似然的相对重要性,将CPE与先验紧密联系起来。我们的研究结果表明,CPE可以出现在隔离合成策展,数据增强,和坏的先验。因此,“在野外”观察到的冷后验不太可能是由单一的简单原因引起的;因此,我们不期望对冷后验进行简单的“修复”。
英文摘要:
The "cold posterior effect" (CPE) in Bayesian deep learning describes the uncomforting observation that the predictive performance of Bayesian neural networks can be significantly improved if the Bayes posterior is artificially sharpened using a temperature parameter T<1. The CPE is problematic in theory and practice and since the effect was identified many researchers have proposed hypotheses to explain the phenomenon. However, despite this intensive research effort the effect remains poorly understood. In this work we provide novel and nuanced evidence relevant to existing explanations for the cold posterior effect, disentangling three hypotheses: 1. The dataset curation hypothesis of Aitchison (2020): we show empirically that the CPE does not arise in a real curated data set but can be produced in a controlled experiment with varying curation strength. 2. The data augmentation hypothesis of Izmailov et al. (2021) and Fortuin et al. (2021): we show empirically that data augmentation is sufficient but not necessary for the CPE to be present. 3. The bad prior hypothesis of Wenzel et al. (2020): we use a simple experiment evaluating the relative importance of the prior and the likelihood, strongly linking the CPE to the prior. Our results demonstrate how the CPE can arise in isolation from synthetic curation, data augmentation, and bad priors. Cold posteriors observed "in the wild" are therefore unlikely to arise from a single simple cause; as a result, we do not expect a simple "fix" for cold posteriors.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。