arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2411.13682cs.LGcs.CRcs.DS
收藏

超越经典维度范式的差分隐私学习

Differentially Private Learning Beyond the Classical Dimensionality Regime

Harvard University(哈佛大学);Rutgers University(罗格斯大学)

Cynthia Dwork, Pranay Tankala, Linjun Zhang

AI总结:

针对比例维度范式($d/n\toδ∈(0,\finity)$)下的差分隐私学习,提出精确误差估计方法,发现类双下降现象并明确不同扰动算法的相对性能,还引入新概率工具提升分析精度。

摘要:

我们首次开展比例维度范式下的差分隐私学习研究,其中数据样本数$n$与问题维度$d$以相互成比例的速率趋近于无穷大,即当$n\to\finity$时,$d/n\toδ$,其中$δ∈(0,\finity)$为任意给定常数。该设置比高维差分隐私学习领域所有先前理论工作的场景更具挑战性——尽管这些工作名为“高维”,但均假设$δ=0$或足够小,对应样本复杂度为$O(d)$的问题,这种范式在现代高维统计中通常被视为“低维”或“经典”。我们针对比例维度范式下的鲁棒线性回归和逻辑回归,提供了几种广泛研究的差分隐私算法(包括输出扰动、目标扰动和噪声随机梯度下降)的误差的精确理论估计。我们误差估计的$1+o(1)$因子精度,比现有更粗糙的分析能更细致地理解这些算法的隐私代价,而现有分析在我们考虑的范式中本质上是无效的。利用这些估计,我们发现了鲁棒线性回归中目标扰动训练误差的一种此前未被观察到的“类双下降”现象。我们还确定了输出扰动平均性能优于目标扰动的场景,反之亦然,证明这些算法的相对性能并不像先前工作所暗示的那样明确。为证明主要定理,我们引入了几种此前未用于分析差分隐私学习算法的概率工具,例如现代高斯比较不等式和起源于统计物理的近期普适性定律。

英文摘要:

We initiate the study of differentially private learning in the proportional dimensionality regime, in which the number of data samples $n$ and problem dimension $d$ approach infinity at rates proportional to one another, meaning that $d/n\toδ$ as $n\to\infty$ for an arbitrary, given constant $δ\in(0,\infty)$. This setting is significantly more challenging than that of all prior theoretical work in high-dimensional differentially private learning, which, despite the name, has assumed that $δ= 0$ or is sufficiently small for problems of sample complexity $O(d)$, a regime typically considered "low-dimensional" or "classical" by modern standards in high-dimensional statistics. We provide sharp theoretical estimates of the error of several well-studied differentially private algorithms for robust linear regression and logistic regression, including output perturbation, objective perturbation, and noisy stochastic gradient descent, in the proportional dimensionality regime. The $1+o(1)$ factor precision of our error estimates enables a far more nuanced understanding of the price of privacy of these algorithms than that afforded by existing, coarser analyses, which are essentially vacuous in the regime we consider. Using our estimates, we discover a previously unobserved "double descent"-like phenomenon in the training error of objective perturbation for robust linear regression. We also identify settings in which output perturbation outperforms objective perturbation on average, and vice versa, demonstrating that the relative performance of these algorithms is less clear-cut than suggested by prior work. To prove our main theorems, we introduce several probabilistic tools that have not previously been used to analyze differentially private learning algorithms, such as a modern Gaussian comparison inequality and recent universality laws with origins in statistical physics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑