arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2003.00505cs.LGcs.AIcs.CR
收藏

具有平滑灵敏度的差分私有深度学习

Differentially Private Deep Learning with Smooth Sensitivity

University of Illinois at Chicago(伊利诺伊大学芝加哥分校);Salesforce Research(Salesforce研究院)

Lichao Sun, Yingbo Zhou, Philip S. Yu, Caiming Xiong

摘要:

确保用于训练现代机器学习模型的敏感数据的隐私性在许多实践领域都至关重要。研究这些问题的一种方法是通过差分隐私的镜头。在此框架中,隐私保证通常通过以使得用于训练模型的数据的细节变得模糊的方式扰动模型来获得。这种方法的一个特定实例是通过“教师-学生”框架,其中拥有敏感数据的教师向学生提供有用但嘈杂的信息,希望允许学生模型在给定任务上表现良好,而无需访问敏感数据的特定特征。因为更强的隐私保障通常涉及教师方面更大的干扰,部署现有的框架从根本上涉及学生的表现和隐私保障之间的权衡。在以前的作品中使用的最重要的技术之一涉及教师模型的合奏,返回信息的基础上嘈杂的投票程序的学生。在这项工作中,我们提出了一种新的投票机制,具有平滑的灵敏度,我们称之为不可变的噪声ArgMax,在一定条件下,可以承受非常大的随机噪声从老师,而不影响有用的信息传递给学生。与以前的工作相比,我们的方法提高了国家的最先进的方法的所有措施,并扩展到更大的任务,更好的性能和更强的隐私($ε\approx 0$)。这个新提出的框架可以应用于任何机器学习模型,并为需要在大量数据上训练的任务提供了一个有吸引力的解决方案。

英文摘要:

Ensuring the privacy of sensitive data used to train modern machine learning models is of paramount importance in many areas of practice. One approach to study these concerns is through the lens of differential privacy. In this framework, privacy guarantees are generally obtained by perturbing models in such a way that specifics of data used to train the model are made ambiguous. A particular instance of this approach is through a "teacher-student" framework, wherein the teacher, who owns the sensitive data, provides the student with useful, but noisy, information, hopefully allowing the student model to perform well on a given task without access to particular features of the sensitive data. Because stronger privacy guarantees generally involve more significant perturbation on the part of the teacher, deploying existing frameworks fundamentally involves a trade-off between student's performance and privacy guarantee. One of the most important techniques used in previous works involves an ensemble of teacher models, which return information to a student based on a noisy voting procedure. In this work, we propose a novel voting mechanism with smooth sensitivity, which we call Immutable Noisy ArgMax, that, under certain conditions, can bear very large random noising from the teacher without affecting the useful information transferred to the student. Compared with previous work, our approach improves over the state-of-the-art methods on all measures, and scale to larger tasks with both better performance and stronger privacy ($ε\approx 0$). This new proposed framework can be applied with any machine learning models, and provides an appealing solution for tasks that requires training on a large amount of data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑