arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2201.12328cs.LG
收藏

基于差分隐私的ImageNet规模训练

Toward Training at ImageNet Scale with Differential Privacy

Google Research(谷歌研究院);Columbia University(哥伦比亚大学)

Alexey Kurakin, Shuang Song, Steve Chien, Roxana Geambasu, Andreas Terzis, Abhradeep Thakurta

摘要:

差分隐私(DP)是训练机器学习(ML)模型(包括神经网络)的事实标准,同时确保训练集中各个示例的隐私。尽管关于如何训练具有差分隐私的ML模型的文献非常丰富,但训练具有合理准确性和隐私性的真实大型神经网络仍然极具挑战性。我们开始研究如何做到这一点,使用ImageNet图像分类作为ML任务的海报示例,该任务目前非常具有挑战性,难以用DP准确解决。本文分享了我们的努力的初步经验教训,希望它将启发和通知其他研究人员探索DP培训的规模。我们展示了有助于使DP训练更快的方法,以及在DP设置中更好地工作的训练过程的模型类型和设置。结合,我们讨论的方法让我们训练Resnet-18与DP到$47.9\%$的准确性和隐私参数$ε= 10, δ= 10^{-6}$。这是对ImageNet模型的“天真”DP训练的重大改进,但与没有隐私的相同网络可以获得的$75\%$准确性相去甚远。我们使用的模型是以Places 365数据集为起点进行预训练的。我们在https://github.com/google-research/dp-imagenet上分享了我们的代码,呼吁其他人在这个新的基础上进一步改进DP。

英文摘要:

Differential privacy (DP) is the de facto standard for training machine learning (ML) models, including neural networks, while ensuring the privacy of individual examples in the training set. Despite a rich literature on how to train ML models with differential privacy, it remains extremely challenging to train real-life, large neural networks with both reasonable accuracy and privacy. We set out to investigate how to do this, using ImageNet image classification as a poster example of an ML task that is very challenging to resolve accurately with DP right now. This paper shares initial lessons from our effort, in the hope that it will inspire and inform other researchers to explore DP training at scale. We show approaches that help make DP training faster, as well as model types and settings of the training process that tend to work better in the DP setting. Combined, the methods we discuss let us train a Resnet-18 with DP to $47.9\%$ accuracy and privacy parameters $ε= 10, δ= 10^{-6}$. This is a significant improvement over "naive" DP training of ImageNet models, but a far cry from the $75\%$ accuracy that can be obtained by the same network without privacy. The model we use was pretrained on the Places365 data set as a starting point. We share our code at https://github.com/google-research/dp-imagenet, calling for others to build upon this new baseline to further improve DP at scale.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑