The privacy preserving properties of Langevin dynamics with additive isotropic noise have been extensively studied. However, the isotropic noise assumption is very restrictive: (a) when adding noise to existing learning algorithms to preserve privacy and maintain the best possible accuracy one should take into account the relative magnitude of the outputs and their correlations; (b) popular algorithms such as stochastic gradient descent (and their continuous time limits) appear to possess anisotropic covariance properties. To study the privacy risks for the anisotropic noise case, one requires general results on the relative entropy between the laws of two Stochastic Differential Equations with different drifts and diffusion coefficients. Our main contribution is to establish such a bound using stability estimates for solutions to the Fokker-Planck equations via functional inequalities. With additional assumptions, the relative entropy bound implies an $(ε,δ)$-differential privacy bound or translates to bounds on the membership inference attack success and we show how anisotropic noise can lead to better privacy-accuracy trade-offs. Finally, the benefits of anisotropic noise are illustrated using numerical results in quadratic loss and neural network setups.
Langevin Unlearning: A New Perspective of Noisy Gradient Descent for Machine Unlearning
Department of Electrical and Computer Engineering（电气与计算机工程系）；Georgia Institute of Technology（佐治亚理工学院）；Department of Mathematics（数学系）；Massachusetts Institute of Technology（麻省理工学院）
Data-dependent PAC-Bayes priors via differential privacy
University of Cambridge（剑桥大学）；Element AI；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
可能近似正确（PAC）贝叶斯框架（McAllester，1999）可以通过使用分布相关先验来整合有关学习算法和（数据）分布的知识，从而在数据相关后验上产生更严格的泛化边界。然而，使用这种灵活性是困难的，特别是当数据分布被假定为未知时。我们展示了ε-差分私有数据依赖先验如何产生有效的PAC-Bayes界，然后展示了选择先验的非私有机制如何也可以产生泛化界。作为该结果的应用，我们证明了通过随机梯度Langevin动力学（SGLD; Welling and Teh，2011）选择的高斯先验均值导致有效的PAC-Bayes界，给定对ε-差分私有平稳分布的2-Wasserstein距离的控制。我们研究我们的数据依赖的界限经验，并表明，他们可以是nonvacuous，即使当其他分布依赖的界限是空的。
Can Microcanonical Langevin Dynamics Leverage Mini-Batch Gradient Noise?
Department of Statistics, LMU Munich（统计系，慕尼黑大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Department of Physics, University of California, Berkeley（伯克利大学物理系）；Department of Astronomy, Tsinghua University（清华大学天文系）；Physics Division, Lawrence Berkeley National Lab（伯克利国家实验室物理部）
Stabilizing Private LASSO under Heterogeneous Covariates via Anisotropic Objective Perturbation
Department of Information Science Ochanomizu University Bunkyo-ku, Tokyo, Japan（信息科学系, 陈仓区, 东京, 日本, 东方大学）；RIKEN center for AIP Chuo-ku, Tokyo, Japan（理化学研究所AIP研究中心, 东京都, 东京, 日本）
Privacy Amplification in Differentially Private Zeroth-Order Optimization with Hidden States
Department of Electrical Engineering, National Taiwan University, Taiwan（台湾国立台湾大学电子工程系）；NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE), Taiwan（国立台湾大学人工智能研究中心（NTU AI-CoRE））；Microsoft, USA（微软公司）；Department of Electrical and Computer Engineering, Georgia Institute of Technology, USA（佐治亚理工学院电子与计算机工程系）
Reducing Diffusion Model Memorization with Higher Order Langevin Dynamics
Department of Applied Math & Statistics（应用数学与统计学系）；Stony Brook University（石溪大学）；Department of Electrical and Computer Engineering（电气与计算机工程系）；Faculty of Engineering（工程学院）；Bar-Ilan University（巴伊兰大学）
Tighter Generalization Bounds for Iterative Differentially Private Learning Algorithms
UBTECH Sydney AI Centre, School of Computer Science, Faculty of Engineering, the University of Sydney（悉尼大学工程学院计算机科学学院优必选悉尼人工智能中心）；University of Science and Technology of China（中国科学技术大学）
