We present backpropagation clipping, a novel variant of differentially private stochastic gradient descent (DP-SGD) for privacy-preserving deep learning. Our approach clips each trainable layer's inputs (during the forward pass) and its upstream gradients (during the backward pass) to ensure bounded global sensitivity for the layer's gradient; this combination replaces the gradient clipping step in existing DP-SGD variants. Our approach is simple to implement in existing deep learning frameworks. The results of our empirical evaluation demonstrate that backpropagation clipping provides higher accuracy at lower values for the privacy parameter $ε$ compared to previous work. We achieve 98.7% accuracy for MNIST with $ε= 0.07$ and 74% accuracy for CIFAR-10 with $ε= 3.64$.
When Gradient Clipping Becomes a Control Mechanism for Differential Privacy in Deep Learning
Computer Science, University of California, Merced, CA 95343, USA（计算机科学，加州大学默塞德分校）；Department of Applied Mathematics, University of California, Merced, CA 95343, USA（应用数学系，加州大学默塞德分校）；Department of Innovation（创新部门）；Research, North Carolina State University, Raleigh, NC, USA（研究，北卡罗来纳州立大学，拉洛城）；Automation (MESA) Lab, Department of Mechanical Engineering, School of Engineering, University of California, Merced, CA 95343, USA（自动化（MESA）实验室，机械工程系，工程学院，加州大学默塞德分校）
University of Minnesota（明尼苏达大学）；Carnegie Mellon University（卡内基梅隆大学）；JD.com, Inc.（京东集团）
Providing privacy protection has been one of the primary motivations of Federated Learning (FL). Recently, there has been a line of work on incorporating the formal privacy notion of differential privacy with FL. To guarantee the client-level differential privacy in FL algorithms, the clients' transmitted model updates have to be clipped before adding privacy noise. Such clipping operation is substantially different from its counterpart of gradient clipping in the centralized differentially private SGD and has not been well-understood. In this paper, we first empirically demonstrate that the c…
Existing approaches for training neural networks with user-level differential privacy (e.g., DP Federated Averaging) in federated learning (FL) settings involve bounding the contribution of each user's model update by clipping it to some constant value. However there is no good a priori setting of the clipping norm across tasks and learning settings: the update norm distribution depends on the model architecture and loss, the amount of data on each device, the client learning rate, and possibly various other parameters. We propose a method wherein instead of a fixed clipping norm, one clips to…
When Differential Privacy Meets Wireless Federated Learning: An Improved Analysis for Privacy and Convergence
School of Informatics, Xiamen University, China（厦门大学信息学院，中国）；Internet of Things Thrust, Hong Kong University of Science and Technology (Guangzhou), China（香港科技大学（广州）物联网研究组，中国）
Penn State University（宾夕法尼亚州立大学）；University of Georgia（佐治亚大学）
Recent work on Renyi Differential Privacy has shown the feasibility of applying differential privacy to deep learning tasks. Despite their promise, however, differentially private deep networks often lag far behind their non-private counterparts in accuracy, showing the need for more research in model architectures, optimizers, etc. One of the barriers to this expanded research is the training time -- often orders of magnitude larger than training non-private networks. The reason for this slowdown is a crucial privacy-related step called "per-example gradient clipping" whose naive implementati…
Large convolutional neural networks (CNN) can be difficult to train in the differentially private (DP) regime, since the optimization algorithms require a computationally expensive operation, known as the per-sample gradient clipping. We propose an efficient and scalable implementation of this clipping on convolutional layers, termed as the mixed ghost clipping, that significantly eases the private training in terms of both time and space complexities, without affecting the accuracy. The improvement in efficiency is rigorously studied through the first complexity analysis for the mixed ghost c…
Taming Noise-Induced Prototype Degradation for Privacy-Preserving Personalized Federated Fine-Tuning
School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；School of Statistics, Renmin University of China（中国人民大学统计学院）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）
DC-SGD: Differentially Private SGD with Dynamic Clipping through Gradient Norm Distribution Estimation
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Ant Group（蚂蚁集团）；School of Engineering, University of Virginia（弗吉尼亚大学工程学院）
Mitigating Disparate Impact of Differentially Private Learning through Bounded Adaptive Clipping
Department of Computer Science, University of Helsinki（计算机科学系，赫尔辛基大学）；Department of Electrical and Computer Engineering, American University of Beirut（电气与计算机工程系，贝鲁特美国大学）
University of Science and Technology of China（中国科学技术大学）；Stanford University（斯坦福大学）；Sun Yat-sen University（中山大学）；Microsoft Research Asia（微软亚洲研究院）；Microsoft Research（微软研究院）
Differentially private deep learning has recently witnessed advances in computational efficiency and privacy-utility trade-off. We explore whether further improvements along the two axes are possible and provide affirmative answers leveraging two instantiations of \emph{group-wise clipping}. To reduce the compute time overhead of private learning, we show that \emph{per-layer clipping}, where the gradient of each neural network layer is clipped separately, allows clipping to be performed in conjunction with backpropagation in differentially private optimization. This results in private learnin…
Double Momentum and Error Feedback for Clipping with Fast Rates and Differential Privacy
University of Basel（巴塞尔大学）；Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（穆罕默德·本·扎耶德人工智能大学（MBZUAI））；King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学（KAUST））
Ant Group（蚂蚁集团）；Shanghai Jiao Tong University（上海交通大学）；University of Minnesota（明尼苏达大学）
The vanilla Differentially-Private Stochastic Gradient Descent (DP-SGD), including DP-Adam and other variants, ensures the privacy of training data by uniformly distributing privacy costs across training steps. The equivalent privacy costs controlled by maintaining the same gradient clipping thresholds and noise powers in each step result in unstable updates and a lower model accuracy when compared to the non-DP counterpart. In this paper, we propose the dynamic DP-SGD (along with dynamic DP-Adam, and others) to reduce the performance loss gap while maintaining privacy by dynamically adjusting…
Training large neural networks with meaningful/usable differential privacy security guarantees is a demanding challenge. In this paper, we tackle this problem by revisiting the two key operations in Differentially Private Stochastic Gradient Descent (DP-SGD): 1) iterative perturbation and 2) gradient clipping. We propose a generic optimization framework, called {\em ModelMix}, which performs random aggregation of intermediate model states. It strengthens the composite privacy analysis utilizing the entropy of the training trajectory and improves the $(ε, δ)$ DP security parameters by an order …
Robust Estimation of Sparse Numerical Vectors under Local Differential Privacy
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Zhejiang University（浙江大学）；Guangzhou University（广州大学）；National Interdisciplinary Research Center of Engineering Physics（国家工程物理学跨学科研究中心）；Anhui University of Science and Technology（安徽理工大学）
We study the difficulties in learning that arise from robust and differentially private optimization. We first study convergence of gradient descent based adversarial training with differential privacy, taking a simple binary classification task on linearly separable data as an illustrative example. We compare the gap between adversarial and nominal risk in both private and non-private settings, showing that the data dimensionality dependent term introduced by private optimization compounds the difficulties of learning a robust model. After this, we discuss what parts of adversarial training a…
University of Amsterdam（阿姆斯特丹大学）；VU Amsterdam（阿姆斯特丹自由大学）
Differentially private learning on real-world data poses challenges for standard machine learning practice: privacy guarantees are difficult to interpret, hyperparameter tuning on private data reduces the privacy budget, and ad-hoc privacy attacks are often required to test model privacy. We introduce three tools to make differentially private machine learning more practical: (1) simple sanity checks which can be carried out in a centralized manner before training, (2) an adaptive clipping bound to reduce the effective number of tuneable privacy parameters, and (3) we show that large-batch tra…
A Differentially Private Kaplan-Meier Estimator for Privacy-Preserving Survival Analysis
Cancer Registry of Norway（挪威癌症登记处）；Norwegian Institute of Public Health（挪威公共卫生研究院）；University of Southern California（南加州大学）；The Arctic University of Norway（挪威北极大学）
Robust Federated Learning Over the Air: Combating Heavy-Tailed Noise with Median Anchored Clipping
ZJU-UIUC Institute, Zhejiang University, Haining 314400, China（浙江大学-UIUC研究院，浙江大学，海宁314400，中国）；Singapore University of Technology and Design（新加坡科技与设计大学）
DP-aware AdaLN-Zero: Taming Conditioning-Induced Heavy-Tailed Gradients in Differentially Private Diffusion
School of Computer Science（计算机科学学院）；Big Data, Minjiang University, Fuzhou, Fujian, China.（大数据，闽江大学，福州，福建，中国）；School of Information, Renmin University of China, Beijing, China（信息学院，中国人民大学，北京，中国）
Deep Learning under Fractional-Order Differential Privacy
MESA Lab, Dept. of EECS University of California, Merced（加州大学默塞德分校电子工程与计算机科学系MESA实验室）；Dept. of Applied Mathematics University of California, Merced（加州大学默塞德分校应用数学系）
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术学院）；Huawei MRC（华为莫斯科研究中心）；Lomonosov MSU（罗蒙诺索夫莫斯科国立大学）
Differential privacy (DP) is a gold-standard concept of measuring and guaranteeing privacy in data analysis. It is well-known that the cost of adding DP to deep learning model is its accuracy. However, it remains unclear how it affects robustness of the model. Standard neural networks are not robust to different input perturbations: either adversarial attacks or common corruptions. In this paper, we empirically observe an interesting trade-off between privacy and robustness of neural networks. We experimentally demonstrate that networks, trained with DP, in some settings might be even more vul…
The application of differential privacy to the training of deep neural networks holds the promise of allowing large-scale (decentralized) use of sensitive data while providing rigorous privacy guarantees to the individual. The predominant approach to differentially private training of neural networks is DP-SGD, which relies on norm-based gradient clipping as a method for bounding sensitivity, followed by the addition of appropriately calibrated Gaussian noise. In this work we propose NeuralDP, a technique for privatising activations of some layer within a neural network, which by the post-proc…
In adaptive dynamic programming, neurocontrol and reinforcement learning, the objective is for an agent to learn to choose actions so as to minimise a total cost function. In this paper we show that when discretized time is used to model the motion of the agent, it can be very important to do "clipping" on the motion of the agent in the final time step of the trajectory. By clipping we mean that the final time step of the trajectory is to be truncated such that the agent stops exactly at the first terminal state reached, and no distance further. We demonstrate that when clipping is omitted, le…
As machine learning becomes more widespread throughout society, aspects including data privacy and fairness must be carefully considered, and are crucial for deployment in highly regulated industries. Unfortunately, the application of privacy enhancing technologies can worsen unfair tendencies in models. In particular, one of the most widely used techniques for private model training, differentially private stochastic gradient descent (DPSGD), frequently intensifies disparate impact on groups within data. In this work we study the fine-grained causes of unfairness in DPSGD and identify gradien…
University of California, Santa Cruz（加州大学圣克鲁兹分校）；Google AI（谷歌人工智能）
Generative Adversarial Networks (GANs) are one of the well-known models to generate synthetic data including images, especially for research communities that cannot use original sensitive datasets because they are not publicly accessible. One of the main challenges in this area is to preserve the privacy of individuals who participate in the training of the GAN models. To address this challenge, we introduce a Differentially Private Conditional GAN (DP-CGAN) training framework based on a new clipping and perturbation strategy, which improves the performance of the model while preserving privac…
We explore the training of generative adversarial networks with differential privacy to anonymize image data sets. On MNIST, we numerically measure the privacy-utility trade-off using parameters from $ε$-$δ$ differential privacy and the inception score. Our experiments uncover a saturated training regime where an increasing privacy budget adds little to the quality of generated images. We also explain analytically why differentially private Adam optimization is independent of the gradient clipping parameter. Furthermore, we highlight common errors in previous works on differentially private de…
LightOn（LightOn公司）；LPENS, École Normale Supérieure（巴黎高等师范学院 LPENS实验室）；Criteo AI Lab（Criteo人工智能实验室）
Optical Processing Units (OPUs) -- low-power photonic chips dedicated to large scale random projections -- have been used in previous work to train deep neural networks using Direct Feedback Alignment (DFA), an effective alternative to backpropagation. Here, we demonstrate how to leverage the intrinsic noise of optical random projections to build a differentially private DFA mechanism, making OPUs a solution of choice to provide a private-by-design training. We provide a theoretical analysis of our adaptive privacy mechanism, carefully measuring how the noise of optical random projections prop…
