Bayesian neural networks (BNNs) have been an important framework in the study of uncertainty quantification. Deterministic variational inference, one of the inference methods, utilizes moment propagation to compute the predictive distributions and objective functions. Unfortunately, deriving the moments requires computationally expensive Taylor expansion in nonlinear functions, such as a rectified linear unit (ReLU) or a sigmoid function. Therefore, a new nonlinear function that realizes faster moment propagation than conventional functions is required. In this paper, we propose a novel nonlinear function named moment propagating-Gaussian error linear unit (MP-GELU) that enables the fast derivation of first and second moments in BNNs. MP-GELU enables the analytical computation of moments by applying nonlinearity to the input statistics, thereby reducing the computationally expensive calculations required for nonlinear functions. In empirical experiments on regression tasks, we observed that the proposed MP-GELU provides higher prediction accuracy and better quality of uncertainty with faster execution than those of ReLU-based BNNs.
Polynomial Composition Activations: Unleashing the Dynamics of Large Language Models
School of Mathematical Sciences, Peking University（北京大学数学科学学院）；Seed-Foundation-Model, ByteDance（字节跳动种子大模型部）；Capital University of Economics and Business（首都经济贸易大学）
National Institute for Fusion Science（日本核融合科学研究所）；The Institute of Statistical Mathematics（统计数理研究所）；Digital Strategy Center, Daicel Corporation（大赛璐公司数字战略中心）；The Graduate University for Advanced Studies (SOKENDAI)（高级研究大学院大学（SOKENDAI））；RIKEN（理化学研究所）
Signal Detection in MIMO Systems with Hardware Imperfections: Message Passing on Neural Networks
Hangzhou Institute of Technology, Xidian University（西安电子科技大学杭州研究院）；Xidian University（西安电子科技大学）；University of Wollongong（伍伦贡大学）；Weizmann Institute of Science（魏茨曼科学研究所）；University of Sydney（悉尼大学）
Uncertainty quantification for noisy inputs-outputs in physics-informed neural networks and neural operators
Brown University（布朗大学）；Huazhong University of Science and Technology（华中科技大学）；Institute of Interdisciplinary Research for Mathematics and Applied Science, School of Mathematics and Statistics, Huazhong University of Science and Technology（华中科技大学数学与统计学院数学与应用科学交叉研究院）
