arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2107.04695cs.LGstat.ML
收藏

L2M:具有优化驱动二阶矩估计的实用后验拉普拉斯近似

L2M: Practical posterior Laplace approximation with optimization-driven second moment estimation

Machine Learning Porto Alegre(阿雷格里港机器学习机构)

Christian S. Perone, Roberto Pereira Silveira, Thomas Paula

摘要:

深度神经网络的不确定性量化最近已经通过许多技术发展。在这项工作中,我们重温拉普拉斯近似,一个经典的后验近似方法,是计算上的吸引力。然而,而不是计算曲率矩阵,我们表明,在一定的正则性条件下,拉普拉斯近似可以很容易地构造使用梯度二阶矩。这个量已经被Adagrad的许多指数移动平均变体(如Adam和RMSprop)估计,但传统上在训练后被丢弃。我们证明了我们的方法(L2M)不需要改变模型或优化,可以在几行代码中实现以产生合理的结果,并且除了优化器已经计算的内容之外,它不需要任何额外的计算步骤,而不需要引入任何新的超参数。我们希望我们的方法可以打开新的研究方向,使用已经计算的数量的优化不确定性估计在深度神经网络。

英文摘要:

Uncertainty quantification for deep neural networks has recently evolved through many techniques. In this work, we revisit Laplace approximation, a classical approach for posterior approximation that is computationally attractive. However, instead of computing the curvature matrix, we show that, under some regularity conditions, the Laplace approximation can be easily constructed using the gradient second moment. This quantity is already estimated by many exponential moving average variants of Adagrad such as Adam and RMSprop, but is traditionally discarded after training. We show that our method (L2M) does not require changes in models or optimization, can be implemented in a few lines of code to yield reasonable results, and it does not require any extra computational steps besides what is already being computed by optimizers, without introducing any new hyperparameter. We hope our method can open new research directions on using quantities already computed by optimizers for uncertainty estimation in deep neural networks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑