模型对称性下的变分推断失效：贝叶斯神经网络的置换不变后验
Variational Inference Failures Under Model Symmetries: Permutation Invariant Posteriors for Bayesian Neural Networks
AI总结：
本研究针对神经网络权重空间置换对称性导致贝叶斯神经网络变分推断后验多模态偏差的问题，提出置换不变变分后验的对称化机制，经理论与实验验证可提升后验拟合效果、预测性能及ELBO值。
摘要：
神经网络架构中的权重空间对称性（例如多层感知机（MLP）中的置换对称性）会使贝叶斯神经网络（Bayesian neural network, BNN）的后验分布具有多个等价模态。这种多模态性给变分推断（variational inference, VI）技术带来了挑战，因为VI通常依赖于用单峰分布来近似后验。在本研究中，我们探究了权重空间置换对称性对VI的影响。我们从理论和实证两方面证明，这些对称性会导致近似后验出现偏差；若不明确加以考虑，这些偏差会降低预测性能和后验拟合效果。为缓解这一问题，我们利用后验的对称结构，设计了一种用于构建置换不变变分后验的对称化机制。我们证明，对称化后的分布对真实后验的拟合效果严格更优，且可通过带有修正KL正则项的原始ELBO目标进行训练。我们通过实验证明，所提方法缓解了上述偏差，并带来了更优的预测结果和更高的ELBO值。
英文摘要：
Weight space symmetries in neural network architectures, such as permutation symmetries in MLPs, give rise to Bayesian neural network (BNN) posteriors with many equivalent modes. This multimodality poses a challenge for variational inference (VI) techniques, which typically rely on approximating the posterior with a unimodal distribution. In this work, we investigate the impact of weight space permutation symmetries on VI. We demonstrate, both theoretically and empirically, that these symmetries lead to biases in the approximate posterior, which degrade predictive performance and posterior fit if not explicitly accounted for. To mitigate this behavior, we leverage the symmetric structure of the posterior and devise a symmetrization mechanism for constructing permutation invariant variational posteriors. We show that the symmetrized distribution has a strictly better fit to the true posterior, and that it can be trained using the original ELBO objective with a modified KL regularization term. We demonstrate experimentally that our approach mitigates the aforementioned biases and results in improved predictions and a higher ELBO.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
