Given the ubiquity of deep neural networks, it is important that these models do not reveal information about sensitive data that they have been trained on. In model inversion attacks, a malicious user attempts to recover the private dataset used to train a supervised neural network. A successful model inversion attack should generate realistic and diverse samples that accurately describe each of the classes in the private dataset. In this work, we provide a probabilistic interpretation of model inversion attacks, and formulate a variational objective that accounts for both diversity and accuracy. In order to optimize this variational objective, we choose a variational family defined in the code space of a deep generative model, trained on a public auxiliary dataset that shares some structural similarity with the target dataset. Empirically, our method substantially improves performance in terms of target attack accuracy, sample realism, and diversity on datasets of faces and chest X-ray images.
Institute for Artificial Intelligence in Medicine（医学人工智能研究所）；Technical University of Munich（慕尼黑工业大学）；Imperial College London（帝国理工学院）
Collaborative machine learning settings like federated learning can be susceptible to adversarial interference and attacks. One class of such attacks is termed model inversion attacks, characterised by the adversary reverse-engineering the model to extract representations and thus disclose the training data. Prior implementations of this attack typically only rely on the captured data (i.e. the shared gradients) and do not exploit the data the adversary themselves control as part of the training consortium. In this work, we propose a novel model inversion framework that builds on the foundatio…
Increasing use of machine learning (ML) technologies in privacy-sensitive domains such as medical diagnoses, lifestyle predictions, and business decisions highlights the need to better understand if these ML technologies are introducing leakage of sensitive and proprietary training data. In this paper, we focus on model inversion attacks where the adversary knows non-sensitive attributes about records in the training data and aims to infer the value of a sensitive attribute unknown to the adversary, using only black-box access to the target classification model. We first devise a novel confide…
Gradient inversion attack (or input recovery from gradient) is an emerging threat to the security and privacy preservation of Federated learning, whereby malicious eavesdroppers or participants in the protocol can recover (partially) the clients' private data. This paper evaluates existing attacks and defenses. We find that some attacks make strong assumptions about the setup. Relaxing such assumptions can substantially weaken these attacks. We then evaluate the benefits of three proposed defense mechanisms against gradient inversion attacks. We show the trade-offs of privacy leakage and data …
Zhejiang University（浙江大学）；Alibaba-Zhejiang University Joint Institute of Frontier Technologies（阿里巴巴-浙江大学前沿技术联合研究院）；Zhejiang Lab（之江实验室）；National University of Singapore（新加坡国立大学）
Neural networks are susceptible to data inference attacks such as the model inversion attack and the membership inference attack, where the attacker could infer the reconstruction and the membership of a data sample from the confidence scores predicted by the target classifier. In this paper, we propose a unified approach, namely purification framework, to defend data inference attacks. It purifies the confidence score vectors predicted by the target classifier by reducing their dispersion. The purifier can be further specialized in defending a particular attack via adversarial learning. We ev…
Increasing use of ML technologies in privacy-sensitive domains such as medical diagnoses, lifestyle predictions, and business decisions highlights the need to better understand if these ML technologies are introducing leakages of sensitive and proprietary training data. In this paper, we focus on one kind of model inversion attacks, where the adversary knows non-sensitive attributes about instances in the training data and aims to infer the value of a sensitive attribute unknown to the adversary, using oracle access to the target classification model. We devise two novel model inversion attrib…
We describe a threat model under which a split network-based federated learning system is susceptible to a model inversion attack by a malicious computational server. We demonstrate that the attack can be successfully performed with limited knowledge of the data distribution by the attacker. We propose a simple additive noise method to defend against model inversion, finding that the method can significantly reduce attack efficacy at an acceptable accuracy trade-off on MNIST. Furthermore, we show that NoPeekNN, an existing defensive method, protects different information from exposure, suggest…
Harvard University（哈佛大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Virginia Tech（弗吉尼亚理工大学）
This paper studies defense mechanisms against model inversion (MI) attacks -- a type of privacy attacks aimed at inferring information about the training data distribution given the access to a target machine learning model. Existing defense mechanisms rely on model-specific heuristics or noise injection. While being able to mitigate attacks, existing methods significantly hinder model performance. There remains a question of how to design a defense mechanism that is applicable to a variety of models and achieves better utility-privacy tradeoff. In this paper, we propose the Mutual Information…
Evaluating Selective Encryption Against Gradient Inversion Attacks
Information Networking Institute（信息网络研究所）；Carnegie Mellon University（卡内基梅隆大学）；TensorOpera AI；Department of Electrical and Computer Engineering（电气与计算机工程系）
Virginia Tech（弗吉尼亚理工大学）；Seattle Research Center, Innopeak Technology（Innopeak Technology 西雅图研究中心）
Model inversion (MI) attacks are aimed at reconstructing training data from model parameters. Such attacks have triggered increasing concerns about privacy, especially given a growing number of online model repositories. However, existing MI attacks against deep neural networks (DNNs) have large room for performance improvement. We present a novel inversion-specific GAN that can better distill knowledge useful for performing attacks on private models from public data. In particular, we train the discriminator to differentiate not only the real and fake samples but the soft-labels provided by t…
University of Science and Technology of China（中国科学技术大学）；Tencent Quantum Lab（腾讯量子实验室）
Many data mining tasks rely on graphs to model relational structures among individuals (nodes). Since relational data are often sensitive, there is an urgent need to evaluate the privacy risks in graph data. One famous privacy attack against data analysis models is the model inversion attack, which aims to infer sensitive data in the training dataset and leads to great privacy concerns. Despite its success in grid-like domains, directly applying model inversion attacks on non-grid domains such as graph leads to poor attack performance. This is mainly due to the failure to consider the unique p…
Breaching FedMD: Image Recovery via Paired-Logits Inversion Attack
The University of Tokyo（东京大学）；Institute for AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
本文发现 FedMD 范式存在隐私泄露风险，提出 PLI 攻击，恶意服务器通过训练反演神经网络利用模型间的置信度差距，仅用配对 logits 即可高成功率重建私有图像。
GI-PIP: Do We Require Impractical Auxiliary Dataset for Gradient Inversion Attacks?
School of Cyber Science and Technology, Beihang University（北京航空航天大学网络空间科学与技术学院）；School of Electronics and Information Engineering, Beihang University（北京航空航天大学电子信息工程学院）
本文提出GI-PIP方法，利用异常检测模型从少量辅助数据中提取图像先验分布，并以异常分数损失调控梯度反转攻击，仅用ImageNet 3.8%的数据即实现16.12 dB PSNR恢复，显著降低了对辅助数据量和分布的要求。
GI-SMN: Gradient Inversion Attack against Federated Learning without Prior Knowledge
College of Information and Technology, Jinan University（暨南大学信息科学技术学院）；Guangdong Key Laboratory of Data Security and Privacy Protection（广东省数据安全与隐私保护重点实验室）；School of Computer Science, Beijing University of Posts and Telecommunications（北京邮电大学计算机学院）
Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）；Universiteit van Amsterdam（阿姆斯特丹大学）
In this work, we explore adversarial attacks on the Variational Autoencoders (VAE). We show how to modify data point to obtain a prescribed latent code (supervised attack) or just get a drastically different code (unsupervised attack). We examine the influence of model modifications ($β$-VAE, NVAE) on the robustness of VAEs and suggest metrics to quantify it.
The Hong Kong Polytechnic University（香港理工大学）；The Hong Kong Polytechnic University Shenzhen Research Institute（香港理工大学深圳研究院）；JD Explore Academy, JD.com（京东探索研究院）
Recent studies have shown that the training samples can be recovered from gradients, which are called Gradient Inversion (GradInv) attacks. However, there remains a lack of extensive surveys covering recent advances and thorough analysis of this issue. In this paper, we present a comprehensive survey on GradInv, aiming to summarize the cutting-edge research and broaden the horizons for different domains. Firstly, we propose a taxonomy of GradInv attacks by characterizing existing attacks into two paradigms: iteration- and recursion-based attacks. In particular, we dig out some critical ingredi…
Fudan University（复旦大学）；University of California at Berkeley（加州大学伯克利分校）；Tsinghua University（清华大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
This paper studies model-inversion attacks, in which the access to a model is abused to infer information about the training data. Since its first introduction, such attacks have raised serious concerns given that training data usually contain privacy-sensitive information. Thus far, successful model-inversion attacks have only been demonstrated on simple models, such as linear regression and logistic regression. Previous attempts to invert neural networks, even the ones with simple architectures, have failed to produce convincing results. We present a novel attack method, termed the generativ…
Zhejiang University（浙江大学）；ZJU-Hangzhou Global Scientific and Technological Innovation Center, Zhejiang University（浙江大学杭州国际科创中心）；Key Laboratory of Blockchain and Cyberspace Governance of Zhejiang Province（浙江省区块链与网络空间治理重点实验室）；Jiaxing Research Institute, Zhejiang University（浙江大学嘉兴研究院）；Zhengzhou Xinda Institute of Advanced Technology（郑州信大先进技术研究院）；National University of Singapore（新加坡国立大学）
Neural networks are susceptible to data inference attacks such as the membership inference attack, the adversarial model inversion attack and the attribute inference attack, where the attacker could infer useful information such as the membership, the reconstruction or the sensitive attributes of a data sample from the confidence scores predicted by the target classifier. In this paper, we propose a method, namely PURIFIER, to defend against membership inference attacks. It transforms the confidence score vectors predicted by the target classifier and makes purified confidence scores indisting…
Towards Eliminating Hard Label Constraints in Gradient Inversion Attacks
School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）
Training deep neural networks often forces users to work in a distributed or outsourced setting, accompanied with privacy concerns. Split learning aims to address this concern by distributing the model among a client and a server. The scheme supposedly provides privacy, since the server cannot see the clients' models and inputs. We show that this is not true via two novel attacks. (1) We show that an honest-but-curious split learning server, equipped only with the knowledge of the client neural network architecture, can recover the input samples and obtain a functionally similar model to the c…
University of Science and Technology of China（中国科学技术大学）；University of Waterloo（滑铁卢大学）；CISPA Helmholtz Center for Information Security（CISPA赫尔姆霍茨信息安全中心）
Model inversion (MI) attacks have raised increasing concerns about privacy, which can reconstruct training data from public models. Indeed, MI attacks can be formalized as an optimization problem that seeks private data in a certain space. Recent MI attacks leverage a generative adversarial network (GAN) as an image prior to narrow the search space, and can successfully reconstruct even the high-dimensional data (e.g., face images). However, these generative MI attacks do not fully exploit the potential capabilities of the target model, still leading to a vague and coupled search space, i.e., …
Federated learning reduces the risk of information leakage, but remains vulnerable to attacks. We investigate how several neural network design decisions can defend against gradients inversion attacks. We show that overlapping gradients provides numerical resistance to gradient inversion on the highly vulnerable dense layer. Specifically, we propose to leverage batching to maximise mixing of gradients by choosing an appropriate loss function and drawing identical labels. We show that otherwise it is possible to directly recover all vectors in a mini-batch without any numerical optimisation due…
Recent works have demonstrated that machine learning models are vulnerable to model inversion attacks, which lead to the exposure of sensitive information contained in their training dataset. While some model inversion attacks have been developed in the past in the black-box attack setting, in which the adversary does not have direct access to the structure of the model, few of these have been conducted so far against complex models such as deep neural networks. In this paper, we introduce GAMIN (for Generative Adversarial Model INversion), a new black-box model inversion attack framework achi…
Recent studies show that the state-of-the-art deep neural networks are vulnerable to model inversion attacks, in which access to a model is abused to reconstruct private training data of any given target class. Existing attacks rely on having access to either the complete target model (whitebox) or the model's soft-labels (blackbox). However, no prior work has been done in the harder but more practical scenario, in which the attacker only has access to the model's predicted label, without a confidence measure. In this paper, we introduce an algorithm, Boundary-Repelling Model Inversion (BREP-M…
University College London（伦敦大学学院）；University of Oxford（牛津大学）；Newcastle University（纽卡斯尔大学）
Many individuals are concerned about the governance of machine learning systems and the prevention of algorithmic harms. The EU's recent General Data Protection Regulation (GDPR) has been seen as a core tool for achieving better governance of this area. While the GDPR does apply to the use of models in some limited situations, most of its provisions relate to the governance of personal data, while models have traditionally been seen as intellectual property. We present recent work from the information security literature around `model inversion' and `membership inference' attacks, which indica…
A New Federated Learning Framework Against Gradient Inversion Attacks
The University of Hong Kong（香港大学）；Beijing University of Technology（北京工业大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；UC Santa Cruz（加州大学圣克鲁兹分校）
Arizona State University（亚利桑那州立大学）；Shanghai Jiao Tong University（上海交通大学）
This work aims to tackle Model Inversion (MI) attack on Split Federated Learning (SFL). SFL is a recent distributed training scheme where multiple clients send intermediate activations (i.e., feature map), instead of raw data, to a central server. While such a scheme helps reduce the computational load at the client end, it opens itself to reconstruction of raw data from intermediate activation by the server. Existing works on protecting SFL only consider inference and do not handle attacks during training. So we propose ResSFL, a Split Federated Learning Framework that is designed to be MI-re…
Fraunhofer AISEC（弗劳恩霍夫人工智能安全研究所）；Technical University Munich（慕尼黑工业大学）
Model inversion (MI) attacks allow to reconstruct average per-class representations of a machine learning (ML) model's training data. It has been shown that in scenarios where each class corresponds to a different individual, such as face classifiers, this represents a severe privacy risk. In this work, we explore a new application for MI: the extraction of speakers' voices from a speaker recognition system. We present an approach to (1) reconstruct audio samples from a trained ML model and (2) extract intermediate voice feature representations which provide valuable insights into the speakers…
Rank Matters: Understanding and Defending Model Inversion Attacks via Low-Rank Feature Filtering
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；South China University of Technology（华南理工大学）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）
Machine learning models often pose a threat to the privacy of individuals whose data is part of the training set. Several recent attacks have been able to infer sensitive information from trained models, including model inversion or attribute inference attacks. These attacks are able to reveal the values of certain sensitive features of individuals who participated in training the model. It has also been shown that several factors can contribute to an increased risk of model inversion, including feature influence. We observe that not all features necessarily share the same level of privacy or …
The successful deployment of artificial intelligence (AI) in many domains from healthcare to hiring requires their responsible use, particularly in model explanations and privacy. Explainable artificial intelligence (XAI) provides more information to help users to understand model decisions, yet this additional knowledge exposes additional risks for privacy attacks. Hence, providing explanation harms privacy. We study this risk for image-based model inversion attacks and identified several attack architectures with increasing performance to reconstruct private image data from model explanation…
University of Sindh（信德大学）；Munster Technological University（芒斯特理工大学）；Nearby Computing（Nearby Computing公司）
The applications concerning vehicular networks benefit from the vision of beyond 5G and 6G technologies such as ultra-dense network topologies, low latency, and high data rates. Vehicular networks have always faced data privacy preservation concerns, which lead to the advent of distributed learning techniques such as federated learning. Although federated learning has solved data privacy preservation issues to some extent, the technique is quite vulnerable to model inversion and model poisoning attacks. We assume that the design of defense mechanism and attacks are two sides of the same coin. …
Technical University of Darmstadt（达姆施塔特工业大学）；Universität der Bundeswehr München（慕尼黑联邦国防军大学）；Centre for Cognitive Science, TU Darmstadt（达姆施塔特工业大学认知科学中心）；Hessian Center for AI (hessian.AI)（黑森人工智能中心（hessian.AI））
Model inversion attacks (MIAs) aim to create synthetic images that reflect the class-wise characteristics from a target classifier's private training data by exploiting the model's learned knowledge. Previous research has developed generative MIAs that use generative adversarial networks (GANs) as image priors tailored to a specific target model. This makes the attacks time- and resource-consuming, inflexible, and susceptible to distributional shifts between datasets. To overcome these drawbacks, we present Plug & Play Attacks, which relax the dependency between the target model and image prio…
