差分隐私（DP）为算法的潜在隐私泄露提供了理论上限，而经验性审计则确立了实际下限。目前已有针对DP训练算法的审计技术。然而机器学习也可以在推理阶段实现私有化。我们提出了首个用于审计私有预测的框架，在该框架中我们实例化了具有不同投毒和查询能力的对手。这使我们能够研究四种私有预测算法的隐私泄露：PATE [Papernot et al., 2016]、CaPC [Choquette-Choo et al al., 2020]、Prompt PATE [Duan et al al., 2023]和Private-kNN [Zhu et al., 2020]。为进行审计，我们引入了新技术，以Renyi DP的经验评估隐私泄露。实验表明：(i)私有预测的隐私分析可以改进，ii)易于投毒的算法导致更高的更高的隐私泄露，iii.没有查询控制权限的对手比比导致具有完全查询控制制的对手。我们的实验表明，我们的实验表明， (i) 私有预测的隐私分析可以改进，(ii) 易于投毒的算法导致更高的隐私泄露，以及 (iii) 对于没有查询控制权的对手，其隐私泄露显著低于具有完全控制权的对手。
英文摘要：
Differential privacy (DP) offers a theoretical upper bound on the potential privacy leakage of analgorithm, while empirical auditing establishes a practical lower bound. Auditing techniques exist forDP training algorithms. However machine learning can also be made private at inference. We propose thefirst framework for auditing private prediction where we instantiate adversaries with varying poisoningand query capabilities. This enables us to study the privacy leakage of four private prediction algorithms:PATE [Papernot et al., 2016], CaPC [Choquette-Choo et al., 2020], PromptPATE [Duan et al., 2023],and Private-kNN [Zhu et al., 2020]. To conduct our audit, we introduce novel techniques to empiricallyevaluate privacy leakage in terms of Renyi DP. Our experiments show that (i) the privacy analysis ofprivate prediction can be improved, (ii) algorithms which are easier to poison lead to much higher privacyleakage, and (iii) the privacy leakage is significantly lower for adversaries without query control than thosewith full control.
Khoury College of Computer Sciences, Northeastern University（东北大学胡里计算机科学学院）
We investigate whether Differentially Private SGD offers better privacy in practice than what is guaranteed by its state-of-the-art analysis. We do so via novel data poisoning attacks, which we show correspond to realistic privacy attacks. While previous work (Ma et al., arXiv 2019) proposed this connection between differential privacy and data poisoning as a defense against data poisoning, our use as a tool for understanding the privacy of a specific mechanism is new. More generally, our work takes a quantitative, empirical approach to understanding the privacy afforded by specific implementa…
Auditing mechanisms for differential privacy use probabilistic means to empirically estimate the privacy level of an algorithm. For private machine learning, existing auditing mechanisms are tight: the empirical privacy estimate (nearly) matches the algorithm's provable privacy guarantee. But these auditing techniques suffer from two limitations. First, they only give tight estimates under implausible worst-case assumptions (e.g., a fully adversarial dataset). Second, they require thousands or millions of training runs to produce non-trivial statistical estimates of the privacy leakage. This w…
Booz Allen Hamilton（博思艾伦汉密尔顿公司）；University of Maryland, Baltimore County（马里兰大学巴尔的摩县分校）；Air Force Research Laboratory（美国空军研究实验室）
We present a framework to statistically audit the privacy guarantee conferred by a differentially private machine learner in practice. While previous works have taken steps toward evaluating privacy loss through poisoning attacks or membership inference, they have been tailored to specific models or have demonstrated low statistical power. Our work develops a general methodology to empirically evaluate the privacy of differentially private machine learning implementations, combining improved privacy search and verification methods with a toolkit of influence-based poisoning attacks. We demonst…
Courant Institute of Mathematical Sciences, New York University（纽约大学库朗数学科学研究所）；IBM Research AI（IBM研究院人工智能部门）
Differential privacy (DP) is the de facto standard for private data release and private machine learning. Auditing black-box DP algorithms and mechanisms to certify whether they satisfy a certain DP guarantee is challenging, especially in high dimension. We propose relaxations of differential privacy based on new divergences on probability distributions: the kernel Rényi divergence and its regularized version. We show that the regularized kernel Rényi divergence can be estimated from samples even in high dimensions, giving rise to auditing procedures for $\varepsilon$-DP, $(\varepsilon,δ)$-DP …
Auditing and Mitigating Privacy Leakage in Cloud-Edge Collaborative Decoding
The Hong Kong Polytechnic University（香港理工大学）；Institute for AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）；School of Software, Tsinghua University（清华大学软件学院）
The Web public key infrastructure is essential to providing secure communication on the Internet today, and certificate authorities play a crucial role in this ecosystem by issuing certificates. These authorities may misissue certificates or suffer misuse attacks, however, which has given rise to the Certificate Transparency (CT) project. The goal of CT is to store all issued certificates in public logs, which can then be checked for the presence of potentially misissued certificates. Thus, the requirement that a given certificate is indeed in one (or several) of these logs lies at the core of…
Machine learning (ML) models have been deployed for high-stakes applications. Due to class imbalance in the sensitive attribute observed in the datasets, ML models are unfair on minority subgroups identified by a sensitive attribute, such as race and sex. In-processing fairness algorithms ensure model predictions are independent of sensitive attribute. Furthermore, ML models are vulnerable to attribute inference attacks where an adversary can identify the values of sensitive attribute by exploiting their distinguishable model predictions. Despite privacy and fairness being important pillars of…
CISPA Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心）；Salesforce Research（Salesforce研究院）
Relying on the fact that not all inputs require the same amount of computation to yield a confident prediction, multi-exit networks are gaining attention as a prominent approach for pushing the limits of efficient deployment. Multi-exit networks endow a backbone model with early exits, allowing to obtain predictions at intermediate layers of the model and thus save computation time and/or energy. However, current various designs of multi-exit networks are only considered to achieve the best trade-off between resource usage efficiency and prediction accuracy, the privacy risks stemming from the…
