arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.03344cs.CR
收藏

重新审视具有先验知识的DP训练中的隐私-效用权衡

Revisiting Privacy-Utility Trade-off for DP Training with Pre-existing Knowledge

Yu Zheng, Wenchao Zhang, Yonggang Zhang, Yuxiang Peng, Wei Song, Kai Zhou, Xiaojiang Du, Bo Han

AI总结:

针对DP-SGD同质噪声损害效用的问题,提出DP-Hero框架,利用先验知识指导异质噪声分配,提升训练准确率,并扩展到联邦训练。

摘要:

差分隐私(DP)通过针对隐私敏感数据集定制随机机制,为保护个体提供了可证明的框架。深度学习模型在模型暴露中已显示出隐私风险,因为已建立的学习模型会无意中记录成员级别的隐私泄露。差分隐私随机梯度下降(DP-SGD)已被提出,通过在反向传播中对梯度更新添加随机高斯噪声来保护训练个体。研究人员发现,DP-SGD会导致效用损失,因为注入的同质噪声可能改变每次迭代中计算的梯度更新。也就是说,梯度中的所有元素都被污染,无论它们在更新模型参数中的重要性如何。在这项工作中,我们认为可以通过引入注入噪声的异质性来优化效用。因此,我们通过定义异质随机机制来抽象其属性,提出了一种具有异质噪声的通用差分隐私框架(DP-Hero)。DP-Hero的洞察在于利用先前训练模型中编码的知识来指导后续噪声异质性的分配,从而利用统计扰动并实现增强的效用。在DP-Hero之上,我们实例化了DP-SGD的异质版本,并将其进一步扩展到联邦训练。我们进行了全面的实验来验证和解释所提出的DP-Hero的有效性,显示与最先进的工作相比,训练准确率有所提高。广泛地说,我们通过从先前训练模型中编码的先前泄露知识中学习噪声指导来改善隐私-效用空间,展示了理解效用改进的DP训练的不同视角。

英文摘要:

Differential privacy (DP) provides a provable framework for protecting individuals by customizing a random mechanism over a privacy-sensitive dataset. Deep learning models have demonstrated privacy risks in model exposure as an established learning model unintentionally records membership-level privacy leakage. Differentially private stochastic gradient descent (DP-SGD) has been proposed to safeguard training individuals by adding random Gaussian noise to gradient updates in the backpropagation. Researchers identify that DP-SGD causes utility loss since the injected homogeneous noise can alter the gradient updates calculated at each iteration. Namely, all elements in the gradient are contaminated regardless of their importance in updating model parameters. In this work, we argue that the utility can be optimized by involving the heterogeneity of the the injected noise. Consequently, we propose a generic differential privacy framework with heterogeneous noise (DP-Hero) by defining a heterogeneous random mechanism to abstract its property. The insight of DP-Hero is to leverage the knowledge encoded in the previously trained model to guide the subsequent allocation of noise heterogeneity, thereby leveraging the statistical perturbation and achieving enhanced utility. Atop DP-Hero, we instantiate a heterogeneous version of DP-SGD, and further extend it to federated training. We conduct comprehensive experiments to verify and explain the effectiveness of the proposed DP-Hero, showing improved training accuracy compared with state-of-the-art works. Broadly, we shed light on improving the privacy-utility space by learning the noise guidance from the pre-existing leaked knowledge encoded in the previously trained model, showing a different perspective of understanding the utility-improved DP training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑