arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2107.04296cs.LGcs.CRcs.CV
收藏

具有Langevin动力学的神经网络的差分私有训练

Differentially private training of neural networks with Langevin dynamics for calibrated predictive uncertainty

Technical University of Munich(慕尼黑工业大学);OpenMined;Imperial College London(帝国理工学院)

Moritz Knolle, Alexander Ziller, Dmitrii Usynin, Rickmer Braren, Marcus R. Makowski, Daniel Rueckert, Georgios Kaissis

摘要:

我们发现,差分私有随机梯度下降(DP-SGD)可能会产生校准不良、过度自信的深度学习模型。这对于安全关键应用(例如,在医疗诊断中)来说是一个严重的问题。我们强调并利用随机梯度朗之万动力学(一种用于训练深度神经网络的可扩展贝叶斯推理技术)与DP-SGD之间的相似之处,以便训练差异私有的贝叶斯神经网络,并对原始(DP-SGD)算法进行微小调整。我们的方法提供了比DP-SGD可靠得多的不确定性估计,如通过减少预期校准误差(MNIST $\sim{5}$ -倍,儿科肺炎数据集$\sim{2}$ -倍)经验证明的那样。

英文摘要:

We show that differentially private stochastic gradient descent (DP-SGD) can yield poorly calibrated, overconfident deep learning models. This represents a serious issue for safety-critical applications, e.g. in medical diagnosis. We highlight and exploit parallels between stochastic gradient Langevin dynamics, a scalable Bayesian inference technique for training deep neural networks, and DP-SGD, in order to train differentially private, Bayesian neural networks with minor adjustments to the original (DP-SGD) algorithm. Our approach provides considerably more reliable uncertainty estimates than DP-SGD, as demonstrated empirically by a reduction in expected calibration error (MNIST $\sim{5}$-fold, Pediatric Pneumonia Dataset $\sim{2}$-fold).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑