发表机构
School of Electrical and Computer Engineering, Georgia Institute of Technology（电气与计算机工程学院，佐治亚理工学院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究在深度神经网络分类器训练中实现端到端差分隐私，提出对训练输入私有化、标签公开的框架，并利用狄利克雷机制和雷尼差分隐私概念，实验表明在多个数据集上达到新最优精度，显著优于先前工作。
AI 中文摘要
差分隐私机器学习能在敏感数据上训练模型，同时确保从模型参数中难以恢复个体数据。现有工作常对训练输入及其标签都进行私有化，而当标签公开或可安全公开时，这些保护可能过于保守。本文提出一种新颖的私有训练框架，对训练输入进行私有化，同时保持标签公开。考虑具有softmax输出层的神经网络，通过应用狄利克雷机制在训练期间随机化softmax输出，以确保训练输入的差分隐私，实现“端到端”标签。利用雷尼差分隐私概念，对狄利克雷机制在重复使用时提供的隐私强度给出严格界限。实验表明，在所有评估的隐私预算下，在CIFAR10、MNIST、MedMNIST、FashionMNIST和SVHN上从头开始训练时，达到了新的最优精度。例如，在CIFAR10上，当实现$(\epsilon, \delta)$-差分隐私且$\delta = 10^{-5}$时，在$\epsilon = 4$时将先前的最优精度从$78.37\%$提高到$88.17\%$，在$\epsilon = 1$时精度为$82.96\%$，显著优于先前工作。
英文摘要
Differentially private machine learning enables model training on sensitive data while ensuring that individual data is unlikely to be recoverable from the parameters of the resulting model. However, existing work often privatizes both training inputs and their labels, and these protections may be conservative when labels are public or can be safely made public. Therefore, in this work we propose a novel private training framework that instead privatizes training inputs while keeping labels public. We consider neural networks with softmax output layers, and thus the mapping from training inputs to the output of the softmax layer is a mapping onto the unit simplex. We randomize softmax outputs during training by applying the Dirichlet mechanism to enforce differential privacy for the training inputs, hence the ``end-to-end'' label. Because training data is reused across multiple training epochs, we use the notion of \Renyi differential privacy to formulate tight bounds on the strength of privacy provided by the Dirichlet mechanism across repeated uses. We show empirically that we attain new state-of-the-art accuracy when training from scratch on CIFAR10, MNIST, MedMNIST, FashionMNIST, and SVHN across all privacy budgets evaluated. Notably, when implementing $(ε, δ)$-differential privacy with $δ=10^{-5}$, we improve the prior state-of-the-art accuracy from $78.37\%$ to $88.17\%$ at $ε=4$ on CIFAR10, and our approach has $82.96\%$ accuracy even for $ε=1$, which significantly outperforms prior work.
Comments32 pages, 10 tables, 3 figures
