arXiv 2102.090302021-02cs.LGmath.OCstat.ML

Proactive DP：一种针对DP-SGD的多目标优化框架

Proactive DP: A Multple Target Optimization Framework for DP-SGD

CWI Amsterdam（阿姆斯特丹数学与计算机科学中心）；Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）；University of Connecticut（康涅狄格大学）；University of Science, Ho Chi Minh（胡志明市科学大学）；Vietnam National University, Ho Chi Minh city（胡志明市越南国家大学）；eBay（eBay公司）；IBM Research（IBM研究院）；Thomas J. Watson Research Center（托马斯·J·沃森研究中心）

Marten van Dijk, Nhuong V. Nguyen, Toan N. Nguyen, Lam M. Nguyen, Phuong Ha Nguyen

备注：: arXiv admin note: text overlap with arXiv:2007.09208, changes in contents and title

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出pro-active DP多目标优化框架，通过改进矩账户方法推导出连接DP-SGD参数的闭式DP保证，并基于此构建效用图与DP计算器，在固定隐私预算下优化模型效用。

摘要：

我们引入了一种针对DP-SGD的多目标优化框架，称为pro-active DP。与用于跟踪隐私预算支出的传统DP账户管理器不同，pro-active DP方案允许人们基于固定的隐私预算（以$ε$和$δ$表示）先验地选择DP-SGD的参数，从而以最优方式最大化预期效用（测试准确率）。为实现这一目标，我们首先对矩账户方法提出了重大改进，提出了一个连接DP-SGD设置中所有参数的闭式$(ε,δ)$-DP保证。我们证明，如果$σ=\sqrt{2(ε+\ln(1/δ))/ε}$，且$T$至少约为$2k^2/ε$，并且$(2/e)^2k^2-1/2\geq \ln(N)$，则DP-SGD是$(ε<0.5,δ=1/N)$-DP的，其中$T$是总轮数，$K=kN$是总梯度计算次数，$k$以大小为$N$的本地数据集的轮数来衡量$K$。我们证明我们的表达式接近紧致，即如果$T$比下界$\approx 2k^2/ε$小一个约4的常数因子，则$(ε,δ)$-DP保证被违反。上述DP保证可以得到增强，即如果$σ= \sqrt{2(ε+\ln(1/δ))/ε}$，且$T$至少约为$2k^2/ε$，再加上两个允许更大$ε\geq 0.5$的、不那么直观的附加条件，则DP-SGD是$(ε, δ)$-DP的。我们的DP理论允许我们创建效用图和DP计算器。这些工具将隐私和效用目标联系起来，并搜索最优的实验设置，有效地兼顾准确性和隐私目标以及实现目标。我们提供了proactive DP的全面实现流程，并通过严格的实验展示了概念验证。

英文摘要：

We introduce a multiple target optimization framework for DP-SGD referred to as pro-active DP. In contrast to traditional DP accountants, which are used to track the expenditure of privacy budgets, the pro-active DP scheme allows one to a-priori select parameters of DP-SGD based on a fixed privacy budget (in terms of $ε$ and $δ$) in such a way to optimize the anticipated utility (test accuracy) the most. To achieve this objective, we first propose significant improvements to the moment account method, presenting a closed-form $(ε,δ)$-DP guarantee that connects all parameters in the DP-SGD setup. We show that DP-SGD is $(ε<0.5,δ=1/N)$-DP if $σ=\sqrt{2(ε+\ln(1/δ))/ε}$ with $T$ at least $\approx 2k^2/ε$ and $(2/e)^2k^2-1/2\geq \ln(N)$, where $T$ is the total number of rounds, and $K=kN$ is the total number of gradient computations where $k$ measures $K$ in number of epochs of size $N$ of the local data set. We prove that our expression is close to tight in that if $T$ is more than a constant factor $\approx 4$ smaller than the lower bound $\approx 2k^2/ε$, then the $(ε,δ)$-DP guarantee is violated. The above DP guarantee can be enhanced in thatDP-SGD is $(ε, δ)$-DP if $σ= \sqrt{2(ε+\ln(1/δ))/ε}$ with $T$ at least $\approx 2k^2/ε$ together with two additional, less intuitive, conditions that allow larger $ε\geq 0.5$. Our DP theory allows us to create a utility graph and DP calculator. These tools link privacy and utility objectives and search for optimal experiment setups, efficiently taking into account both accuracy and privacy objectives, as well as implementation goals. We furnish a comprehensive implementation flow of our proactive DP, with rigorous experiments to showcase the proof-of-concept.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。