arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2202.03770cs.LGcs.AI
收藏

参数稀疏性对贝叶斯深度学习的随机梯度MCMC方法的影响

Impact of Parameter Sparsity on Stochastic Gradient MCMC Methods for Bayesian Deep Learning

University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校);SRI International(斯坦福国际研究院);US Army Research Laboratory(美国陆军研究实验室)

Meet P. Vadera, Adam D. Cobb, Brian Jalaian, Benjamin M. Marlin

摘要:

贝叶斯方法对于提高深度神经网络模型的不确定性量化能力和鲁棒性具有重要的前景。最近的研究已经看到了一些深度神经网络的近似贝叶斯推理方法的调查,建立在变分贝叶斯和马尔可夫链蒙特卡罗(MCMC)框架。MCMC方法的一个基本问题是,它们所实现的改进是以增加计算时间和模型存储成本为代价的。在本文中,我们研究了稀疏网络结构的潜力,灵活地权衡模型存储成本和推理运行时间对预测性能和不确定性量化能力。我们使用随机梯度MCMC方法作为核心贝叶斯推理方法,并考虑了各种方法来选择稀疏网络结构。令人惊讶的是,我们的研究结果表明,某些类别的随机选择的子结构可以执行以及来自最先进的迭代修剪方法的子结构,同时大大减少模型训练时间。

英文摘要:

Bayesian methods hold significant promise for improving the uncertainty quantification ability and robustness of deep neural network models. Recent research has seen the investigation of a number of approximate Bayesian inference methods for deep neural networks, building on both the variational Bayesian and Markov chain Monte Carlo (MCMC) frameworks. A fundamental issue with MCMC methods is that the improvements they enable are obtained at the expense of increased computation time and model storage costs. In this paper, we investigate the potential of sparse network structures to flexibly trade-off model storage costs and inference run time against predictive performance and uncertainty quantification ability. We use stochastic gradient MCMC methods as the core Bayesian inference method and consider a variety of approaches for selecting sparse network structures. Surprisingly, our results show that certain classes of randomly selected substructures can perform as well as substructures derived from state-of-the-art iterative pruning methods while drastically reducing model training times.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑