DPAdapter：通过噪声容忍预训练改进差分隐私深度学习
DPAdapter: Improving Differentially Private Deep Learning through Noise Tolerance Pre-training
AI总结：
针对差分隐私导致深度学习模型性能下降的问题，提出即插即用的DPAdapter技术，通过改进SAM算法的双批次策略提升参数抗噪鲁棒性，在隐私预算ε=4时将SOTA DPML算法的平均准确率从72.92%提升至77.09%。
摘要：
近期研究进展凸显了差分隐私（differential privacy, DP）在保护机器学习模型训练所用个人数据方面的关键作用。然而，由于训练过程中引入的扰动，整合DP往往会导致模型性能显著下降，这成为差分隐私机器学习（differentially private machine learning, DPML）领域的一项严峻挑战。为此，已有多项缓解方案被提出，通常围绕构建新的DPML算法或放宽DP定义以适配不同场景展开。尽管已有这些举措，DP对模型（尤其是大规模模型）造成的性能衰减仍然十分显著，因此需要一种创新方案，能够巧妙规避DP对模型效用的实质性损害。\n针对这一问题，我们提出DPAdapter——一项旨在通过提升参数鲁棒性来增强DPML算法模型性能的开创性技术。该策略的核心直觉是：具备鲁棒参数的模型天生对DP引入的噪声具有更强的抵抗能力，从而在存在扰动的情况下仍能保持更优性能。DPAdapter对锐度感知最小化（sharpness-aware minimization, SAM）技术进行了改进与增强，采用双批次策略以提供更准确的扰动估计和高效的梯度下降，从而提升参数对噪声的鲁棒性。值得注意的是，DPAdapter可作为即插即用组件，与现有DPML算法结合以进一步提升其性能。我们的实验表明，DPAdapter大幅提升了当前最优的DPML算法性能，在隐私预算ε=4的条件下，将平均准确率从72.92%提升至77.09%。
英文摘要：
Recent developments have underscored the critical role of \textit{differential privacy} (DP) in safeguarding individual data for training machine learning models. However, integrating DP oftentimes incurs significant model performance degradation due to the perturbation introduced into the training process, presenting a formidable challenge in the {differentially private machine learning} (DPML) field. To this end, several mitigative efforts have been proposed, typically revolving around formulating new DPML algorithms or relaxing DP definitions to harmonize with distinct contexts. In spite of these initiatives, the diminishment induced by DP on models, particularly large-scale models, remains substantial and thus, necessitates an innovative solution that adeptly circumnavigates the consequential impairment of model utility. In response, we introduce DPAdapter, a pioneering technique designed to amplify the model performance of DPML algorithms by enhancing parameter robustness. The fundamental intuition behind this strategy is that models with robust parameters are inherently more resistant to the noise introduced by DP, thereby retaining better performance despite the perturbations. DPAdapter modifies and enhances the sharpness-aware minimization (SAM) technique, utilizing a two-batch strategy to provide a more accurate perturbation estimate and an efficient gradient descent, thereby improving parameter robustness against noise. Notably, DPAdapter can act as a plug-and-play component and be combined with existing DPML algorithms to further improve their performance. Our experiments show that DPAdapter vastly enhances state-of-the-art DPML algorithms, increasing average accuracy from 72.92\% to 77.09\% with a privacy budget of $ε=4$.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
