arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.05564cs.LGcs.ARstat.ML
收藏

模拟贝叶斯神经网络对权重分布形状不敏感

Analog Bayesian neural networks are insensitive to the shape of the weight distribution

Sandia National Laboratories(桑迪亚国家实验室)

Ravi G. Patel, T. Patrick Xiao, Sapan Agarwal, Christopher Bennett

AI总结:

针对模拟硬件难以精确控制 MFVI 噪声分布形状的问题,本文提出直接使用真实器件噪声作为变分分布训练 BNN,并证明预测分布仅取决于权重均值与方差。

摘要:

近期研究表明,采用平均场变分推理(MFVI)训练的贝叶斯神经网络(BNN)可在模拟硬件上实现,与标准数字实现相比有望带来数个数量级的能耗节省。然而,尽管 MFVI 中通常使用高斯分布作为变分分布,但要精确控制模拟器件采样所产生噪声分布的形状却十分困难。本文提出一种利用真实器件噪声作为变分分布进行 MFVI 训练的方法。此外,我们通过实验证明:对于具有相同权重均值和方差的 BNN,无论变分分布形状如何,其预测分布都会收敛到同一分布。该结果表明,模拟器件设计者在硬件实现执行 MFVI 的 BNN 时,无需考虑器件噪声分布的形状。

英文摘要:

Recent work has demonstrated that Bayesian neural networks (BNN's) trained with mean field variational inference (MFVI) can be implemented in analog hardware, promising orders of magnitude energy savings compared to the standard digital implementations. However, while Gaussians are typically used as the variational distribution in MFVI, it is difficult to precisely control the shape of the noise distributions produced by sampling analog devices. This paper introduces a method for MFVI training using real device noise as the variational distribution. Furthermore, we demonstrate empirically that the predictive distributions from BNN's with the same weight means and variances converge to the same distribution, regardless of the shape of the variational distribution. This result suggests that analog device designers do not need to consider the shape of the device noise distribution when hardware-implementing BNNs performing MFVI.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑