arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.01864cs.LGcs.CR
收藏

回收碎片:利用中间检查点改进私有学习

Recycling Scraps: Improving Private Learning by Leveraging Intermediate Checkpoints

Google(谷歌)

Virat Shejwalkar, Arun Ganesh, Rajiv Mathews, Yarong Mu, Shuang Song, Om Thakkar, Abhradeep Thakurta, Xinyi Zheng

AI总结:

本研究提出利用训练中间检查点聚合的通用框架,在不增加隐私成本的前提下提升差分隐私机器学习的准确率与方差表现,并探索了DP模型预测方差的估计方法。

摘要:

在本研究中,我们聚焦于优化前沿差分隐私机器学习(DP ML)方法的准确率-方差权衡。首先,我们设计了一个通用框架,该框架利用训练过程中中间检查点的聚合结果来提升DP ML技术的准确率。具体而言,我们证明,在StackOverflow、CIFAR10和CIFAR100数据集上,基于聚合结果的训练相比现有前沿方法能显著提升预测准确率。例如,我们将前沿DP StackOverflow的准确率提升至:ε=8.2时为22.74%(相对提升2.06%),ε=18.9时为23.90%(相对提升2.09%)。此外,在训练数据分布周期性变化的场景下,这些提升效果会进一步放大。我们还证明,在一项专有的生产级pCVR任务中,我们的方法在效用和方差方面分别实现了0.54%和62.6%的相对提升。最后,我们率先探索了DP噪声给DP ML模型预测带来的不确定性(方差)的估计方法。我们证明,在损失函数的标准假设下,最后几个检查点的样本方差可以很好地近似DP运行最终模型的方差。实验结果表明,最后几个检查点可以为收敛的DP模型的方差提供合理的下界。关键在于,本文提出的所有方法都基于DP ML技术的单次训练运行,因此不会产生额外的隐私成本。

英文摘要:

In this work, we focus on improving the accuracy-variance trade-off for state-of-the-art differentially private machine learning (DP ML) methods. First, we design a general framework that uses aggregates of intermediate checkpoints \emph{during training} to increase the accuracy of DP ML techniques. Specifically, we demonstrate that training over aggregates can provide significant gains in prediction accuracy over the existing state-of-the-art for StackOverflow, CIFAR10 and CIFAR100 datasets. For instance, we improve the state-of-the-art DP StackOverflow accuracies to 22.74\% (+2.06\% relative) for $ε=8.2$, and 23.90\% (+2.09\%) for $ε=18.9$. Furthermore, these gains magnify in settings with periodically varying training data distributions. We also demonstrate that our methods achieve relative improvements of 0.54\% and 62.6\% in terms of utility and variance, on a proprietary, production-grade pCVR task. Lastly, we initiate an exploration into estimating the uncertainty (variance) that DP noise adds in the predictions of DP ML models. We prove that, under standard assumptions on the loss function, the sample variance from last few checkpoints provides a good approximation of the variance of the final model of a DP run. Empirically, we show that the last few checkpoints can provide a reasonable lower bound for the variance of a converged DP model. Crucially, all the methods proposed in this paper operate on \emph{a single training run} of the DP ML technique, thus incurring no additional privacy cost.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑