arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.16163cs.LGcs.AIstat.ML
收藏

神经网络中协方差传播的解析解

An Analytic Solution to Covariance Propagation in Neural Networks

Carnegie Mellon University(卡内基梅隆大学)

Oren Wright, Yorie Nakahira, José M. F. Moura

AI总结:

本文提出一种无样本矩传播技术,通过推导非线性激活函数协方差的解析解,精确表征神经网络的输入-输出分布,提升了深度学习系统不确定性量化的准确性与效率。

摘要:

神经网络的不确定性量化对于衡量深度学习系统的可靠性和鲁棒性至关重要。然而,这通常涉及成本高昂或不准确的采样方法与近似。本文提出了一种无样本矩传播技术,在网络中传播均值向量和协方差矩阵,以准确表征神经网络的输入-输出分布。该技术的关键支撑是针对通过非线性激活函数(如Heaviside、ReLU和GELU)的随机变量协方差的解析解。实验分析了已训练神经网络的输入-输出分布并训练了Bayesian neural networks,展示了所提技术的广泛适用性与优势。

英文摘要:

Uncertainty quantification of neural networks is critical to measuring the reliability and robustness of deep learning systems. However, this often involves costly or inaccurate sampling methods and approximations. This paper presents a sample-free moment propagation technique that propagates mean vectors and covariance matrices across a network to accurately characterize the input-output distributions of neural networks. A key enabler of our technique is an analytic solution for the covariance of random variables passed through nonlinear activation functions, such as Heaviside, ReLU, and GELU. The wide applicability and merits of the proposed technique are shown in experiments analyzing the input-output distributions of trained neural networks and training Bayesian neural networks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑