Characterizing Internal Evasion Attacks in Federated Learning
Carnegie Mellon University（卡内基梅隆大学）
Taejin Kim, Shubhranshu Singh, Nikhil Madaan, Carlee Joe-Wong
备注：
16 pages, 8 figures (14 images if counting sub-figures separately), Camera ready version for AISTATS 2023, longer version of paper submitted to CrossFL 2022 poster workshop, code available at (https://github.com/tj-kim/pFedDef_v1)
Federated learning allows for clients in a distributed system to jointly train a machine learning model. However, clients' models are vulnerable to attacks during the training and testing phases. In this paper, we address the issue of adversarial clients performing "internal evasion attacks": crafting evasion attacks at test time to deceive other clients. For example, adversaries may aim to deceive spam filters and recommendation systems trained with federated learning for monetary gain. The adversarial clients have extensive information about the victim model in a federated learning setting, as weight information is shared amongst clients. We are the first to characterize the transferability of such internal evasion attacks for different learning methods and analyze the trade-off between model accuracy and robustness depending on the degree of similarities in client data. We show that adversarial training defenses in the federated learning setting only display limited improvements against internal attacks. However, combining adversarial training with personalized federated learning frameworks increases relative internal attack robustness by 60% compared to federated adversarial training and performs well under limited system resources.
Federated learning enables thousands of participants to construct a deep learning model without sharing their private training data with each other. For example, multiple smartphones can jointly train a next-word predictor for keyboards without revealing what individual users type. We demonstrate that any participant in federated learning can introduce hidden backdoor functionality into the joint global model, e.g., to ensure that an image classifier assigns an attacker-chosen label to images with certain features, or that a word predictor completes certain sentences with an attacker-chosen wo…
Federated learning (FL) is an effective mechanism for data privacy in recommender systems by running machine learning model training on-device. While prior FL optimizations tackled the data and system heterogeneity challenges faced by FL, they assume the two are independent of each other. This fundamental assumption is not reflective of real-world, large-scale recommender systems -- data and system heterogeneity are tightly intertwined. This paper takes a data-driven approach to show the inter-dependence of data and system heterogeneity in real-world data and quantifies its impact on the overa…
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Norton Research Group（诺顿研究集团）
Evasion attack in multi-label learning systems is an interesting, widely witnessed, yet rarely explored research topic. Characterizing the crucial factors determining the attackability of the multi-label adversarial threat is the key to interpret the origin of the adversarial vulnerability and to understand how to mitigate it. Our study is inspired by the theory of adversarial risk bound. We associate the attackability of a targeted multi-label classifier with the regularity of the classifier and the training data distribution. Beyond the theoretical attackability analysis, we further propose …
Key Lab of High Confidence Software Technologies (Peking University)（高可信软件技术重点实验室（北京大学））；State Key Laboratory of Networking and Switching Technology (BUPT)（网络与交换技术国家重点实验室（北京邮电大学））；Department of Computer Science and Technology, Peking University（北京大学计算机科学技术系）；Microsoft Research（微软研究院）
Federated learning (FL) is an emerging, privacy-preserving machine learning paradigm, drawing tremendous attention in both academia and industry. A unique characteristic of FL is heterogeneity, which resides in the various hardware specifications and dynamic states across the participating devices. Theoretically, heterogeneity can exert a huge influence on the FL training process, e.g., causing a device unavailable for training or unable to upload its model updates. Unfortunately, these impacts have never been systematically studied and quantified in existing FL literature. In this paper, we c…
Imperial College London（帝国理工学院）；IBM Research（IBM研究院）
Federated learning (FL) is one of the most important paradigms addressing privacy and data governance issues in machine learning (ML). Adversarial training has emerged, so far, as the most promising approach against evasion threats on ML models. In this paper, we take the first known steps towards federated adversarial training (FAT) combining both methods to reduce the threat of evasion during inference while preserving the data privacy during training. We investigate the effectiveness of the FAT protocol for idealised federated settings using MNIST, Fashion-MNIST, and CIFAR10, and provide fi…
We study whether iterated vector fields (vector fields composed with themselves) are conservative. We give explicit examples of vector fields for which this self-composition preserves conservatism. Notably, this includes gradient vector fields of loss functions associated with some generalized linear models. As we show, characterizing the set of vector fields satisfying this condition leads to non-trivial geometric questions. In the context of federated learning, we show that when clients have loss functions whose gradients satisfy this condition, federated averaging is equivalent to gradient …
Localization and tracking of objects using data-driven methods is a popular topic due to the complexity in characterizing the physics of wireless channel propagation models. In these modeling approaches, data needs to be gathered to accurately train models, at the same time that user's privacy is maintained. An appealing scheme to cooperatively achieve these goals is known as Federated Learning (FL). A challenge in FL schemes is the presence of non-independent and identically distributed (non-IID) data, caused by unevenly exploration of different areas. In this paper, we consider the use of re…
Queen’s University（女王大学）；Pukyong National University（釜庆国立大学）
Federated Learning (FL) has recently received a lot of attention for large-scale privacy-preserving machine learning. However, high communication overheads due to frequent gradient transmissions decelerate FL. To mitigate the communication overheads, two main techniques have been studied: (i) local update of weights characterizing the trade-off between communication and computation and (ii) gradient compression characterizing the trade-off between communication and precision. To the best of our knowledge, studying and balancing those two trade-offs jointly and dynamically while considering the…
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Google Research（谷歌研究院）
While recent works have indicated that federated learning (FL) may be vulnerable to poisoning attacks by compromised clients, their real impact on production FL systems is not fully understood. In this work, we aim to develop a comprehensive systemization for poisoning attacks on FL by enumerating all possible threat models, variations of poisoning, and adversary capabilities. We specifically put our focus on untargeted poisoning attacks, as we argue that they are significantly relevant to production FL deployments. We present a critical analysis of untargeted poisoning attacks under practical…
Federated learning (FL) is becoming a popular paradigm for collaborative learning over distributed, private datasets owned by non-trusting entities. FL has seen successful deployment in production environments, and it has been adopted in services such as virtual keyboards, auto-completion, item recommendation, and several IoT applications. However, FL comes with the challenge of performing training over largely heterogeneous datasets, devices, and networks that are out of the control of the centralized FL server. Motivated by this inherent setting, we make a first step towards characterizing t…
The Pennsylvania State University（宾夕法尼亚州立大学）；The University of Texas at Austin（得克萨斯大学奥斯汀分校）
In federated learning, communication cost is often a critical bottleneck to scale up distributed optimization algorithms to collaboratively learn a model from millions of devices with potentially unreliable or limited communication and heterogeneous data distributions. Two notable trends to deal with the communication overhead of federated algorithms are gradient compression and local computation with periodic communication. Despite many attempts, characterizing the relationship between these two approaches has proven elusive. We address this by proposing a set of algorithms with periodical co…
Toyota Technological Institute at Chicago（芝加哥丰田理工学院）；University of California, Berkeley（加州大学伯克利分校）；Cornell University（康奈尔大学）
In recent years, federated learning has been embraced as an approach for bringing about collaboration across large populations of learning agents. However, little is known about how collaboration protocols should take agents' incentives into account when allocating individual resources for communal learning in order to maintain such collaborations. Inspired by game theoretic notions, this paper introduces a framework for incentive-aware learning and data sharing in federated learning. Our stable and envy-free equilibria capture notions of collaboration in the presence of agents interested in m…
University of Nevada, Reno（内华达大学里诺分校）；IBM Research（IBM研究院）；University of Virginia（弗吉尼亚大学）
Data heterogeneity has been identified as one of the key features in federated learning but often overlooked in the lens of robustness to adversarial attacks. This paper focuses on characterizing and understanding its impact on backdooring attacks in federated learning through comprehensive experiments using synthetic and the LEAF benchmarks. The initial impression driven by our experimental results suggests that data heterogeneity is the dominant factor in the effectiveness of attacks and it may be a redemption for defending against backdooring as it makes the attack less efficient, more chal…
Fed-GraB: Federated Long-tailed Learning with Self-Adjusting Gradient Balancer
Zhejiang University（浙江大学）；The Hong Kong University of Science and Technology（香港科技大学）；Angelalign Technology Inc（时代天使科技有限公司）；State Key Laboratory of Oral Diseases, Sichuan University（四川大学口腔疾病研究国家重点实验室）；Singapore University of Technology and Design（新加坡科技设计大学）
Convergence-Latency-Aware Adaptive Modulation and Resource Allocation in RIS-Assisted Wireless Federated Learning
Broadband Access Network Laboratory, Shanghai Jiao Tong University（宽带接入网络实验室，上海交通大学）；School of Information Science and Engineering, Southeast University（信息科学与工程学院，东南大学）；Data61, CSIRO（数据61，联邦科学与工业研究组织）；Department of Electronic and Electrical Engineering, University College London（电子与电气工程系，伦敦大学学院）；School of Internet of Things Engineering, Jiangnan University（物联网工程学院，江南大学）
OptiVote: Non-Coherent FSO Over-the-Air Majority Vote for Communication-Efficient Distributed Federated Learning in Space Data Centers
School of Control Science and Engineering, Shandong University（控制科学与工程学院，山东大学）；Department of Computer Science, University of Exeter（计算机科学系，埃克塞特大学）；Department of ECE, The Hong Kong University of Science and Technology（电子与电气工程系，香港科学与技术大学）；School of Electrical Engineering, Chongqing University（电气工程学院，重庆大学）；Information Systems Technology and Design Pillar, Singapore University of Technology and Design（信息系统技术与设计学院，新加坡科技设计大学）
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Inria（法国国家信息与自动化研究所）
Despite of the pervasive existence of multi-label evasion attack, it is an open yet essential problem to characterize the origin of the adversarial vulnerability of a multi-label learning system and assess its attackability. In this study, we focus on non-targeted evasion attack against multi-label classifiers. The goal of the threat is to cause miss-classification with respect to as many labels as possible, with the same input perturbation. Our work gains in-depth understanding about the multi-label adversarial attack by first characterizing the transferability of the attack based on the func…
Federated Learning for Surgical Vision in Appendicitis Classification: Results of the FedSurg EndoVis 2024 Challenge
Department of Translational Surgical Oncology, National Center for Tumor Diseases (NCT), NCT/UCC Dresden, a partnership between DKFZ, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology（转化外科肿瘤学部，肿瘤疾病国家中心（NCT），NCT/UCC德累斯顿，DKFZ、医学院和卡尔·戈斯瓦尔德·卡尔医院之间的合作）；Centre for Tactile Internet with Human-in-the-Loop (CeTI), TUD Dresden University of Technology（人机协同触觉互联网中心（CeTI），德累斯顿技术大学）；Department of Medicine I, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology（医学部I，医学院和卡尔·戈斯瓦尔德·卡尔医院，德累斯顿技术大学）；Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg（肿瘤医学部，肿瘤疾病国家中心（NCT），海德堡大学医院）；Weldon School of Biomedical Engineering, Purdue University（生物医学工程学院，普渡大学）；Else Kroener Fresenius Center for Digital Health, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology（Else Kroener Fresenius数字健康中心，医学院和卡尔·戈斯瓦尔德·卡尔医院，德累斯顿技术大学）；IHU Strasbourg, Institute of Image-Guided Surgery（斯特拉斯堡IHU，影像引导手术研究所）；University of Strasbourg, CNRS, INSERM, ICube, UMR7357（斯特拉斯堡大学，CNRS，INSERM，ICube，UMR7357）；Dr. NTR University of Health Sciences（Dr. NTR健康科学大学）；Department of Visceral, Thoracic and Vascular Surgery, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology（visceral、胸腔和血管外科部，医学院和卡尔·戈斯瓦尔德·卡尔医院，德累斯顿技术大学）；Digital Technologies, Medtronic（数字技术，美敦力）；UCL Hawkes Institute and Department of Computer Science, University College London（UCL Hawkes研究所和计算机科学系，伦敦大学学院）；German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems（德国癌症研究中心（DKFZ）海德堡，智能医学系统部）；German Cancer Research Center (DKFZ) Heidelberg, Helmholtz Imaging（德国癌症研究中心（DKFZ）海德堡，海德堡成像）；National Center for Tumor Diseases (NCT) Heidelberg, a partnership between DKFZ and Heidelberg University Hospital（肿瘤疾病国家中心（NCT）海德堡，DKFZ和海德堡大学医院之间的合作）；Faculty of Mathematics and Computer Science, Heidelberg University（数学和计算机科学学院，海德堡大学）；Helmholtz Information and Data Science School for Health（海德堡信息与数据科学健康学院）；Heidelberg University Hospital, Surgical Clinic, Surgical AI Research Group（海德堡大学医院，外科诊所，外科AI研究组）；Krankenhaus St. Joseph-Stift Dresden GmbH（德累斯顿圣约瑟夫修道院医院）
