Federated Learning (FL) has been widely accepted as the solution for privacy-preserving machine learning without collecting raw data. While new technologies proposed in the past few years do evolve the FL area, unfortunately, the evaluation results presented in these works fall short in integrity and are hardly comparable because of the inconsistent evaluation metrics and experimental settings. In this paper, we propose a holistic evaluation framework for FL called FedEval, and present a benchmarking study on seven state-of-the-art FL algorithms. Specifically, we first introduce the core evaluation taxonomy model, called FedEval-Core, which covers four essential evaluation aspects for FL: Privacy, Robustness, Effectiveness, and Efficiency, with various well-defined metrics and experimental settings. Based on the FedEval-Core, we further develop an FL evaluation platform with standardized evaluation settings and easy-to-use interfaces. We then provide an in-depth benchmarking study between the seven well-known FL algorithms, including FedSGD, FedAvg, FedProx, FedOpt, FedSTC, SecAgg, and HEAgg. We comprehensively analyze the advantages and disadvantages of these algorithms and further identify the suitable practical scenarios for different algorithms, which is rarely done by prior work. Lastly, we excavate a set of take-away insights and future research directions, which are very helpful for researchers in the FL area.
ZKP-FedEval: Verifiable and Privacy-Preserving Federated Evaluation using Zero-Knowledge Proofs
Department of Multidisciplinary Engineering, Texas A&M University（多学科工程系，德克萨斯A&M大学）；Department of Computer Science, Ho Technical University（计算机科学系，霍技术大学）；Department of Computer Engineering, Kwame Nkrumah University of Science and Technology (KNUST)（计算机工程系，库马西科学与技术大学（KNUST））
Holistic Evaluation Metrics: Use Case Sensitive Evaluation Metrics for Federated Learning
School of Electrical and Computer Engineering, The University of Sydney（悉尼大学电气与计算机工程学院）；Department of Information Systems and Cyber Security, University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校信息系统与网络安全系）
Federated data analytics is a framework for distributed data analysis where a server compiles noisy responses from a group of distributed low-bandwidth user devices to estimate aggregate statistics. Two major challenges in this framework are privacy, since user data is often sensitive, and compression, since the user devices have low network bandwidth. Prior work has addressed these challenges separately by combining standard compression algorithms with known privacy mechanisms. In this work, we take a holistic look at the problem and design a family of privacy-aware compression mechanisms tha…
Even though recent years have seen many attacks exposing severe vulnerabilities in Federated Learning (FL), a holistic understanding of what enables these attacks and how they can be mitigated effectively is still lacking. In this work, we demystify the inner workings of existing (targeted) attacks. We provide new insights into why these attacks are possible and why a definitive solution to FL robustness is challenging. We show that the need for ML algorithms to memorize tail data has significant implications for FL integrity. This phenomenon has largely been studied in the context of privacy;…
Andalusian Research Institute in Data Science and Computational Intelligence, University of Granada（格拉纳达大学安达卢西亚数据科学与计算智能研究所）
The high demand of artificial intelligence services at the edges that also preserve data privacy has pushed the research on novel machine learning paradigms that fit those requirements. Federated learning has the ambition to protect data privacy through distributed learning methods that keep the data in their data silos. Likewise, differential privacy attains to improve the protection of data privacy by measuring the privacy loss in the communication among the elements of federated learning. The prospective matching of federated learning and differential privacy to the challenges of data priva…
Tsinghua University（清华大学）；Fraunhofer Heinrich Hertz Institute（弗劳恩霍夫海因里希·赫兹研究所）；RWTH Aachen University（亚琛工业大学）；A*STAR（新加坡科技研究局）；Keio University（庆应义塾大学）；Technical University of Berlin（柏林工业大学）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所）
The advent of Federated Learning (FL) has ignited a new paradigm for parallel and confidential decentralized Machine Learning (ML) with the potential of utilizing the computational power of a vast number of IoT, mobile and edge devices without data leaving the respective device, ensuring privacy by design. Yet, in order to scale this new paradigm beyond small groups of already entrusted entities towards mass adoption, the Federated Learning Framework (FLF) has to become (i) truly decentralized and (ii) participants have to be incentivized. This is the first systematic literature review analyzi…
FedEx: Expediting Federated Learning over Heterogeneous Mobile Devices by Overlapping and Participant Selection
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications（网络与交换技术国家重点实验室，北京邮电大学）；College of Electronic Information Engineering, Taiyuan University of Technology（电子信息工程学院，太原科技大学）；Department of Advanced Interdisciplinary Research, Pengcheng Laboratory（先进跨学科研究部，鹏城实验室）；Department of Electrical and Computer Engineering, University of Houston（电气与计算机工程系，德克萨斯大学）
FedGCS: A Generative Framework for Efficient Client Selection in Federated Learning via Gradient-based Optimization
Computer Network Information Center, Chinese Academy of Sciences（中国科学院计算机网络信息中心）；University of Chinese Academy of Sciences（中国科学院大学）；Hangzhou Institute for Advanced Study, UCAS（UCAS杭州先进研究院）；Department of Computer and Information Science, IOTSC, University of Macau（澳门科技大学计算机与信息科学系）；University of Oxford（牛津大学）
Federated Learning with Differential Privacy: An Utility-Enhanced Approach
Deakin University（迪肯大学）；Data61, CSIRO（澳大利亚联邦科学与工业研究组织Data61）；The University of Alabama in Huntsville（阿拉巴马大学亨茨维尔分校）；University of New South Wales (UNSW)（新南威尔士大学）
Communication-Efficient Multimodal Federated Learning: Joint Modality and Client Selection
School of Electrical and Computer Engineering, Purdue University（普渡大学电气与计算机工程学院）；Department of Computer Science and Engineering, Yonsei University（延世大学计算机科学与工程系）；School of Electrical and Computer Engineering, Princeton University（普林斯顿大学电气与计算机工程学院）；Saab Inc.（Saab公司）
FedLF: Adaptive Logit Adjustment and Feature Optimization in Federated Long-Tailed Learning
School of Cyber Science and Engineering, Qufu Normal University（曲阜师范大学网络空间安全学院）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
KTH Royal Institute of Technology（瑞典皇家理工学院）；Nanyang Technological University（南洋理工大学）；City University of Hong Kong（香港城市大学）；National University of Singapore（新加坡国立大学）
Choice Outweighs Effort: Facilitating Complementary Knowledge Fusion in Federated Learning via Re-calibration and Merit-discrimination
Key Lab. of Computing Power Network；Information Security, Ministry of Education, Shandong Computer Science Center, Qilu University of Technology (Shandong Academy of Sciences), Jinan, China；Shandong Provincial Key Lab. of Industrial Network；Information System Security, Shandong Fundamental Research Center for Computer Science, Jinan, China；College of Control Science；Engineering, Zhejiang University, Hangzhou, China
By focusing on immersive interaction among users, the burgeoning Metaverse can be viewed as a natural extension of existing social media. Similar to traditional online social networks, there are numerous security and privacy issues in the Metaverse (e.g., attacks on user authentication and impersonation). In this paper, we develop a holistic research agenda for zero-trust user authentication in social virtual reality (VR), an early prototype of the Metaverse. Our proposed research includes four concrete steps: investigating biometrics-based authentication that is suitable for continuously auth…
The University of Danang - Vietnam-Korea University of Information and Communication Technology（岘港大学-越韩信息与通信技术大学）；Kyung Hee University（庆熙大学）；Aalborg University（奥尔堡大学）；The University of Sydney（悉尼大学）
Emerging cross-device artificial intelligence (AI) applications require a transition from conventional centralized learning systems towards large-scale distributed AI systems that can collaboratively perform complex learning tasks. In this regard, democratized learning (Dem-AI) lays out a holistic philosophy with underlying principles for building large-scale distributed and democratized machine learning systems. The outlined principles are meant to study a generalization in distributed learning systems that goes beyond existing mechanisms such as federated learning. Moreover, such learning sy…
