Federated learning (FL) is a popular technique to train machine learning (ML) models on decentralized data sources. In order to sustain long-term participation of data owners, it is important to fairly appraise each data source and compensate data owners for their contribution to the training process. The Shapley value (SV) defines a unique payoff scheme that satisfies many desiderata for a data value notion. It has been increasingly used for valuing training data in centralized learning. However, computing the SV requires exhaustively evaluating the model performance on every subset of data sources, which incurs prohibitive communication cost in the federated setting. Besides, the canonical SV ignores the order of data sources during training, which conflicts with the sequential nature of FL. This paper proposes a variant of the SV amenable to FL, which we call the federated Shapley value. The federated SV preserves the desirable properties of the canonical SV while it can be calculated without incurring extra communication cost and is also able to capture the effect of participation order on data value. We conduct a thorough empirical study of the federated SV on a range of tasks, including noisy label detection, adversarial participant detection, and data summarization on different benchmark datasets, and demonstrate that it can reflect the real utility of data sources for FL and has the potential to enhance system robustness, security, and efficiency. We also report and analyze "failure cases" and hope to stimulate future research.
University of British Columbia（不列颠哥伦比亚大学）；Huawei Technologies Canada Co.（华为技术加拿大公司）；Simon Fraser University（西蒙弗雷泽大学）；KTH Royal Institute of Technology（KTH皇家理工学院）
Federated learning is an emerging decentralized machine learning scheme that allows multiple data owners to work collaboratively while ensuring data privacy. The success of federated learning depends largely on the participation of data owners. To sustain and encourage data owners' participation, it is crucial to fairly evaluate the quality of the data provided by the data owners and reward them correspondingly. Federated Shapley value, recently proposed by Wang et al. [Federated Learning, 2020], is a measure for data value under the framework of federated learning that satisfies many desired …
Principled Federated Random Forests for Heterogeneous Data
Inria（法国国家信息与自动化研究所）；PreMeDICaL（预医学实验室）；Inserm（法国国家医学研究院）；University of Montpellier（蒙彼利埃大学）；Sorbonne Université（索邦大学）；Université Paris Cité（巴黎城市大学）；CNRS（国家科学研究中心）；LPSM（巴黎高等师范学院）
Massachusetts General Hospital（麻省总医院）；Institute for Infocomm Research, A*STAR（新加坡科技研究局资讯通信研究院）
Federated Learning (FL) wherein multiple institutions collaboratively train a machine learning model without sharing data is becoming popular. Participating institutions might not contribute equally, some contribute more data, some better quality data or some more diverse data. To fairly rank the contribution of different institutions, Shapley value (SV) has emerged as the method of choice. Exact SV computation is impossibly expensive, especially when there are hundreds of contributors. Existing SV computation techniques use approximations. However, in healthcare where the number of contributi…
Data Valuation and Selection in a Federated Model Marketplace
IORA National University of Singapore（国际研究机构国立新加坡大学）；NUS Research Institution（国立新加坡大学研究机构）；Department of ECE（电子工程系）；Virginia Tech（弗吉尼亚理工大学）；Business School（商学院）
Efficient Data Valuation Approximation in Federated Learning: A Sampling-based Approach
State Key Laboratory of Complex & Critical Software Environment Lab, School of Computer Science and Engineering, Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University, China（复杂与关键软件环境实验室，计算机科学与工程学院，未来区块链与隐私计算先进创新中心，北京航空航天大学）
Theoretically Principled Federated Learning for Balancing Privacy and Utility
Huazhong University of Science and Technology（华中科技大学）；Tsinghua University（清华大学）；Peng Cheng Laboratory（鹏城实验室）；WeBank（微众银行）；The Hong Kong Polytechnic University（香港理工大学）
Huazhong University of Science and Technology（华中科技大学）；University of Waterloo（滑铁卢大学）；Northeastern University（东北大学）；KTH Royal Institute of Technology（瑞典皇家理工学院）
Traditional machine learning relies on a centralized data pipeline, i.e., data are provided to a central server for model training. In many applications, however, data are inherently fragmented. Such a decentralized nature of these databases presents the biggest challenge for collaboration: sending all decentralized datasets to a central server raises serious privacy concerns. Although there has been a joint effort in tackling such a critical issue by proposing privacy-preserving machine learning frameworks, such as federated learning, most state-of-the-art frameworks are built still in a cent…
Federated Bayesian learning offers a principled framework for the definition of collaborative training algorithms that are able to quantify epistemic uncertainty and to produce trustworthy decisions. Upon the completion of collaborative training, an agent may decide to exercise her legal "right to be forgotten", which calls for her contribution to the jointly trained model to be deleted and discarded. This paper studies federated learning and unlearning in a decentralized network within a Bayesian framework. It specifically develops federated variational inference (VI) solutions based on the d…
In federated optimization, heterogeneity in the clients' local datasets and computation speeds results in large variations in the number of local updates performed by each client in each communication round. Naive weighted aggregation of such models causes objective inconsistency, that is, the global model converges to a stationary point of a mismatched objective function which can be arbitrarily different from the true objective. This paper provides a general framework to analyze the convergence of federated heterogeneous optimization algorithms. It subsumes previously proposed methods such a…
Ping An Technology (Shenzhen) Co., Ltd.（平安科技（深圳）有限公司）
In federated learning (FL), fair and accurate measurement of the contribution of each federated participant is of great significance. The level of contribution not only provides a rational metric for distributing financial benefits among federated participants, but also helps to discover malicious participants that try to poison the FL framework. Previous methods for contribution measurement were based on enumeration over possible combination of federated participants. Their computation costs increase drastically with the number of participants or feature dimensions, making them inapplicable i…
Federated learning is typically approached as an optimization problem, where the goal is to minimize a global loss function by distributing computation across client devices that possess local data and specify different parts of the global objective. We present an alternative perspective and formulate federated learning as a posterior inference problem, where the goal is to infer a global posterior distribution by having client devices each infer the posterior of their local data. While exact inference is often intractable, this perspective provides a principled way to search for global optima…
Client contribution evaluation, also known as data valuation, is a crucial approach in federated learning(FL) for client selection and incentive allocation. However, due to restrictions of accessibility of raw data, only limited information such as local weights and local data size of each client is open for quantifying the client contribution. Using data size from available information, we introduce an empirical evaluation method called Federated Client Contribution Evaluation through Accuracy Approximation(FedCCEA). This method builds the Accuracy Approximation Model(AAM), which estimates a …
Turku University of Applied Sciences（图尔库应用科学大学）；Riphah International University（里法哈国际大学）
In federated learning (FL), the global model at the server requires an efficient mechanism for weight aggregation and a systematic strategy for collaboration selection to manage and optimize communication payload. We introduce a practical and cost-efficient method for regularized weight aggregation and propose a laborsaving technique to select collaborators per round. We illustrate the performance of our method, regularized similarity weight aggregation (RegSimAgg), on the Federated Tumor Segmentation (FeTS) 2022 challenge's federated training (weight aggregation) problem. Our scalable approac…
Task Diversity in Bayesian Federated Learning: Simultaneous Processing of Classification and Regression
School of Statistics, Renmin University of China（中国人民大学统计学院）；School of Statistics and Data Science, Southeast University（东南大学统计与数据科学学院）；Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing（未来区块链与隐私计算北京高精尖创新中心）
Federated Learning (FL) is emerging as a promising paradigm of privacy-preserving machine learning, which trains an algorithm across multiple clients without exchanging their data samples. Recent works highlighted several privacy and robustness weaknesses in FL and addressed these concerns using local differential privacy (LDP) and some well-studied methods used in conventional ML, separately. However, it is still not clear how LDP affects adversarial robustness in FL. To fill this gap, this work attempts to develop a comprehensive understanding of the effects of LDP on adversarial robustness …
Federated learning (FL) is an emerging distributed machine learning framework for collaborative model training with a network of clients (edge devices). FL offers default client privacy by allowing clients to keep their sensitive data on local devices and to only share local training parameter updates with the federated server. However, recent studies have shown that even sharing local parameter updates from a client to the federated server may be susceptible to gradient leakage attacks and intrude the client privacy regarding its training data. In this paper, we present a principled framework…
Inria（法国国家信息与自动化研究所）；Université Côte d’Azur（蔚蓝海岸大学）；Accenture Labs（埃森哲实验室）；Université de Lille（里尔大学）
The increasing size of data generated by smartphones and IoT devices motivated the development of Federated Learning (FL), a framework for on-device collaborative training of machine learning models. First efforts in FL focused on learning a single global model with good average performance across clients, but the global model may be arbitrarily bad for a given client, due to the inherent heterogeneity of local data distributions. Federated multi-task learning (MTL) approaches can learn personalized models by formulating an opportune penalized optimization problem. The penalization term can ca…
FlatLand: Personalized Graph Federated Learning via Tailored Lorentz Space
The Chinese University of Hong Kong（香港中文大学）；Yale University（耶鲁大学）；Huawei Technologies Co., Ltd.（华为技术有限公司）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Naval Information Warfare Center Pacific（太平洋海军信息战中心）
In this work, we explore combining automatic hyperparameter tuning and optimization for federated learning (FL) in an online, one-shot procedure. We apply a principled approach on a method for adaptive client learning rate, number of local steps, and batch size. In our federated learning applications, our primary motivations are minimizing communication budget as well as local computational resources in the training pipeline. Conventionally, hyperparameter tuning methods involve at least some degree of trial-and-error, which is known to be sample inefficient. In order to address our motivation…
Owen Sampling Accelerates Contribution Estimation in Federated Learning
Department of Electrical and Software Engineering, University of Calgary, Calgary, Alberta, Canada（电气与软件工程系，卡里尔大学）；Department of Electrical Engineering and Computer Science, York University, Toronto, Ontario, Canada（电气工程与计算机科学系，约克大学）；School of Information, University of Michigan, Ann Arbor, Michigan, USA（信息学院，密歇根大学）
Fairness-Aware Federated Learning with Trajectory Shapley Value
Faculty of Mathematics, Ecole Normale Supérieure Paris-Saclay（巴黎-萨克雷大学数学系）；Chair for Dynamics, Control, Machine Learning and Numerics – Alexander von Humboldt Professorship, Department of Mathematics, Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔兰根-纽伦堡弗里德里希-亚历山大大学数学系）
Overcoming label shift with target-aware federated learning
RISE Research Institutes of Sweden（瑞典RISE研究机构）；KTH Royal Institute of Technology（皇家理工学院）；Chalmers University of Technology（查尔姆斯理工大学）；University of Gothenburg（哥德堡大学）
