隐私保护的盲模型平均 - 分布式、非交互式且收敛
Private Blind Model Averaging - Distributed, Non-interactive, and Convergent
AI总结：
本文提出盲模型平均方法，通过非交互式输出扰动实现分布式差分隐私学习，在多类分类任务中提升隐私与效用的权衡。
摘要：
分布式差分隐私学习技术使大量用户能够共同学习一个模型，而无需首先集中收集训练数据。同时，用户之间的通信以及最终得到的模型不应泄露有关训练数据的信息。这种学习技术可以部署在边缘设备上，如果能够扩展到大量用户，特别是如果通信减少到最低限度：无交互，即每个参与方只发送一条消息。此前已知的最佳方法基于梯度平均，这本质上需要许多同步轮次。一种有希望的非交互式替代方案依赖于所谓的输出扰动：每个用户首先在本地完成训练，然后提交其模型进行安全平均，而无需进一步同步。我们分析了这种范式，我们称之为盲模型平均（BlindAvg），在凸和平滑的经验风险最小化（ERM）设置中，例如支持向量机（SVM）。虽然所需的噪声尺度在渐近意义上与集中设置相同，但不清楚BlindAvg接近集中学习的程度，即其效用成本。我们通过两个贡献来表征并提升BlindAvg的隐私-效用权衡：首先，我们证明，对于非光滑SVM学习者，足够强的L2正则化使BlindAvg趋向于集中设置。其次，我们引入了新的差分隐私凸和平滑ERM学习器SoftmaxReg，在多类设置中比SVM具有更好的隐私-效用权衡。我们在三个数据集（CIFAR-10、CIFAR-100和联邦EMNIST）上评估了我们的发现，并在人工极端非独立同分布（non-IID）场景中提供了消融分析。
英文摘要：
Distributed differentially private learning techniques enable a large number of users to jointly learn a model without having to first centrally collect the training data. At the same time, neither the communication between the users nor the resulting model shall leak information about the training data. This kind of learning technique can be deployed to edge devices if it can be scaled up to a large number of users, particularly if the communication is reduced to a minimum: no interaction, i.e., each party only sends a single message. The best previously known methods are based on gradient averaging, which inherently requires many synchronization rounds. A promising non-interactive alternative to gradient averaging relies on so-called output perturbation: each user first locally finishes training and then submits its model for secure averaging without further synchronization. We analyze this paradigm, which we coin blind model averaging (BlindAvg), in the setting of convex and smooth empirical risk minimization (ERM) like a support vector machine (SVM). While the required noise scale is asymptotically the same as in the centralized setting, it is not well understood how close BlindAvg comes to centralized learning, i.e., its utility cost. We characterize and boost the privacy-utility tradeoff of BlindAvg with two contributions: First, we prove that BlindAvg converges towards the centralized setting for a sufficiently strong L2-regularization for a non-smooth SVM learner. Second, we introduce the novel differentially private convex and smooth ERM learner SoftmaxReg that has a better privacy-utility tradeoff than an SVM in a multi-class setting. We evaluate our findings on three datasets (CIFAR-10, CIFAR-100, and Federated EMNIST) and provide an ablation in an artificially extreme non-IID scenario.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
