Secure aggregation is a critical component in federated learning (FL), which enables the server to learn the aggregate model of the users without observing their local models. Conventionally, secure aggregation algorithms focus only on ensuring the privacy of individual users in a single training round. We contend that such designs can lead to significant privacy leakages over multiple training rounds, due to partial user selection/participation at each round of FL. In fact, we show that the conventional random user selection strategies in FL lead to leaking users' individual models within number of rounds that is linear in the number of users. To address this challenge, we introduce a secure aggregation framework, Multi-RoundSecAgg, with multi-round privacy guarantees. In particular, we introduce a new metric to quantify the privacy guarantees of FL over multiple training rounds, and develop a structured user selection strategy that guarantees the long-term privacy of each user (over any number of training rounds). Our framework also carefully accounts for the fairness and the average number of participating users at each round. Our experiments on MNIST and CIFAR-10 datasets in the IID and the non-IID settings demonstrate the performance improvement over the baselines, both in terms of privacy protection and test accuracy.
Hong Kong University of Science & Technology（香港科技大学）
In federated learning, multiple parties collaborate in order to train a global model over their respective datasets. Even though cryptographic primitives (e.g., homomorphic encryption) can help achieve data privacy in this setting, some partial information may still be leaked across parties if this is done non-judiciously. In this work, we study the federated learning framework of SecureBoost [Cheng et al., FL@IJCAI'19] as a specific such example, demonstrate a leakage-abuse attack based on its leakage profile, and experimentally evaluate the effectiveness of our attack. We then propose two se…
Game-Theoretic Machine Unlearning: Mitigating Extra Privacy Leakage
School of Information Engineering, Zhongnan University of Economics and Law（中南财经政法大学信息工程学院）；Faculty of Data Science, City University of Macau（澳门城市大学数据科学学院）
Petrus team, Inria, France（法国国家信息与自动化研究所 Petrus 团队）；Magnet team, Inria, France（法国国家信息与自动化研究所 Magnet 团队）
Imagine a group of citizens willing to collectively contribute their personal data for the common good to produce socially useful information, resulting from data analytics or machine learning computations. Sharing raw personal data with a centralized server performing the computation could raise concerns about privacy and a perceived risk of mass surveillance. Instead, citizens may trust each other and their own devices to engage into a decentralized computation to collaboratively produce an aggregate data release to be shared. In the context of secure computing nodes exchanging messages over…
Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；University of Waterloo（滑铁卢大学）
Federated Learning (FL) is a prominent framework that enables training a centralized model while securing user privacy by fusing local, decentralized models. In this setting, one major obstacle is data heterogeneity, i.e., each client having non-identically and independently distributed (non-IID) data. This is analogous to the context of Domain Generalization (DG), where each client can be treated as a different domain. However, while many approaches in DG tackle data heterogeneity from the algorithmic perspective, recent evidence suggests that data augmentation can induce equal or greater per…
Concept bottleneck models perform classification by first predicting which of a list of human provided concepts are true about a datapoint. Then a downstream model uses these predicted concept labels to predict the target label. The predicted concepts act as a rationale for the target prediction. Model trust issues emerge in this paradigm when soft concept labels are used: it has previously been observed that extra information about the data distribution leaks into the concept predictions. In this work we show how Monte-Carlo Dropout can be used to attain soft concept predictions that do not c…
A large body of work shows that machine learning (ML) models can leak sensitive or confidential information about their training data. Recently, leakage due to distribution inference (or property inference) attacks is gaining attention. In this attack, the goal of an adversary is to infer distributional information about the training data. So far, research on distribution inference has focused on demonstrating successful attacks, with little attention given to identifying the potential causes of the leakage and to proposing mitigations. To bridge this gap, as our main contribution, we theoreti…
Image recognition systems have demonstrated tremendous progress over the past few decades thanks, in part, to our ability of learning compact and robust representations of images. As we witness the wide spread adoption of these systems, it is imperative to consider the problem of unintended leakage of information from an image representation, which might compromise the privacy of the data owner. This paper investigates the problem of learning an image representation that minimizes such leakage of user information. We formulate the problem as an adversarial non-zero sum game of finding a good e…
Mitigating Membership Inference in Intermediate Representations with Differentially Private Training
School of Information, Renmin University of China, Beijing, China（中国人民大学信息学院，北京，中国）；School of Computer Science（计算机科学学院）；Big Data, Minjiang University, Fuzhou, Fujian, China（大数据，闽江大学，福州，福建，中国）
Vanderbilt University（范德堡大学）；University of Virginia（弗吉尼亚大学）；Case Western Reserve University（凯斯西储大学）
Graph Neural Networks (GNNs) have shown great power in learning node representations on graphs. However, they may inherit historical prejudices from training data, leading to discriminatory bias in predictions. Although some work has developed fair GNNs, most of them directly borrow fair representation learning techniques from non-graph domains without considering the potential problem of sensitive attribute leakage caused by feature propagation in GNNs. However, we empirically observe that feature propagation could vary the correlation of previously innocuous non-sensitive features to the sen…
Securing Autonomous Vehicle Systems via Twin-Aware Federated Reinforcement Learning
North Carolina State University（北卡罗来纳州立大学）；University of Louisville（路易斯维尔大学）；University of Maryland（马里兰大学）；University of North Texas（北德克萨斯大学）；Wayne State University（韦恩州立大学）
Securing LLMs in the Wild: Privacy and Security Challenges at the Edge
Department of Electrical and Computer Engineering, University of South Florida（电子与计算机工程系，佛罗里达州立大学）；Department of Mathematics, Embry-Riddle Aeronautical University（数学系，埃默里-瑞德航空大学）
University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；University of Washington（华盛顿大学）
Voice assistants are deployed widely and provide useful functionality. However, recent work has shown that commercial systems like Amazon Alexa and Google Home are vulnerable to voice-based confusion attacks that exploit design issues. We propose a systems-oriented defense against this class of attacks and demonstrate its functionality for Amazon Alexa. We ensure that only the skills a user intends execute in response to voice commands. Our key insight is that we can interpret a user's intentions by analyzing their activity on counterpart systems of the web and smartphones. For example, the Ly…
Adaptive Learning for IRS-Assisted Wireless Networks: Securing Opportunistic Communications Against Byzantine Eavesdroppers
Department of Electrical Engineering, Columbia University（哥伦比亚大学电气工程系）；Department of Electrical Engineering, Imam Khomeini International University（伊斯兰教法国际大学电气工程系）；Department of Electrical Engineering, Qatar University（卡塔尔大学电气工程系）
Princeton University（普林斯顿大学）；National University of Singapore（新加坡国立大学）
The arms race between attacks and defenses for machine learning models has come to a forefront in recent years, in both the security community and the privacy community. However, one big limitation of previous research is that the security domain and the privacy domain have typically been considered separately. It is thus unclear whether the defense methods in one domain will have any unexpected impact on the other domain. In this paper, we take a step towards resolving this limitation by combining the two domains. In particular, we measure the success of membership inference attacks against s…
The proliferation of smart, connected, always listening devices have introduced significant privacy risks to users in a smart home environment. Beyond the notable risk of eavesdropping, intruders can adopt machine learning techniques to infer sensitive information from audio recordings on these devices, resulting in a new dimension of privacy concerns and attack variables to smart home users. Techniques such as sound masking and microphone jamming have been effectively used to prevent eavesdroppers from listening in to private conversations. In this study, we explore the problem of adversaries…
Speaker recognition systems (SRSs) have recently been shown to be vulnerable to adversarial attacks, raising significant security concerns. In this work, we systematically investigate transformation and adversarial training based defenses for securing SRSs. According to the characteristic of SRSs, we present 22 diverse transformations and thoroughly evaluate them using 7 recent promising adversarial attacks (4 white-box and 3 black-box) on speaker recognition. With careful regard for best practices in defense evaluations, we analyze the strength of transformations to withstand adaptive attacks…
No Prior, No Leakage: Revisiting Reconstruction Attacks in Trained Neural Networks
Department of Computer Science Cranberry-Lemon University（计算机科学系克兰伯瑞-柠檬大学）；Department of Computational Neuroscience University of the Witwatersrand（计算神经科学系沃图瓦兹兰大学）；Tel Aviv University（特拉维夫大学）；Ben-Gurion University of the Negev（贝内-乔尔大学）；Bar-Ilan University（巴-伊兰大学）
