The emerging availability of trained machine learning models has put forward the novel concept of Machine Learning Model Market in which one can harness the collective intelligence of multiple well-trained models to improve the performance of the resultant model through one-shot federated learning and ensemble learning in a data-free manner. However, picking the models available in the market for ensemble learning is time-consuming, as using all the models is not always the best approach. It is thus crucial to have an effective ensemble selection strategy that can find a good subset of the base models for the ensemble. Conventional ensemble selection techniques are not applicable, as we do not have access to the local datasets of the parties in the federated learning setting. In this paper, we present a novel Data-Free Diversity-Based method called DeDES to address the ensemble selection problem for models generated by one-shot federated learning in practical applications such as model markets. Experiments showed that our method can achieve both better performance and higher efficiency over 5 datasets and 4 different model structures under the different data-partition strategies.
Data Valuation and Selection in a Federated Model Marketplace
IORA National University of Singapore（国际研究机构国立新加坡大学）；NUS Research Institution（国立新加坡大学研究机构）；Department of ECE（电子工程系）；Virginia Tech（弗吉尼亚理工大学）；Business School（商学院）
Federated Learning Incentive Mechanism under Buyers' Auction Market
University of Electronic Science and Technology of China（电子科技大学）；Beihang University（北京航空航天大学）；National Taiwan University（台湾大学）；Jiangxi University of Finance and Economics（江西财经大学）
University of Science and Technology of China (USTC)（中国科学技术大学）；Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）
Attributes skew hinders the current federated learning (FL) frameworks from consistent optimization directions among the clients, which inevitably leads to performance reduction and unstable convergence. The core problems lie in that: 1) Domain-specific attributes, which are non-causal and only locally valid, are indeliberately mixed into global aggregation. 2) The one-stage optimizations of entangled attributes cannot simultaneously satisfy two conflicting objectives, i.e., generalization and personalization. To cope with these, we proposed disentangled federated learning (DFL) to disentangle…
Fair Federated Learning under Domain Skew with Local Consistency and Domain Diversity
National Engineering Research Center for Multimedia Software（国家多媒体软件工程技术研究中心）；School of Computer Science, Wuhan University（武汉大学计算机学院）；Taikang Center for Life and Medical Sciences, Wuhan University（武汉大学泰康生命医学中心）
FedFusion: Federated Learning with Diversity- and Cluster-Aware Encoders for Robust Adaptation under Label Scarcity
Department of Computer Science University of the Western Cape（计算机科学系乌得勒支大学）；Department of Computer Science Missouri University of Science and Technology（计算机科学系密苏里科技大学）
Federated Learning for Data Market: Shapley-UCB for Seller Selection and Incentives
School of Intelligent Systems Engineering, Sun Yat-Sen University（中山大学智能系统工程学院）；Department of Computing, The Hong Kong Polytechnic University（香港理工大学计算机系）
Federated learning (FL) is rapidly gaining popularity and enables multiple data owners ({\em a.k.a.} FL participants) to collaboratively train machine learning models in a privacy-preserving way. A key unaddressed scenario is that these FL participants are in a competitive market, where market shares represent their competitiveness. Although they are interested to enhance the performance of their respective models through FL, market leaders (who are often data owners who can contribute significantly to building high performance FL models) want to avoid losing their market shares by enhancing t…
KAIST AI（韩国科学技术院人工智能研究院）；NAVER AI LAB（NAVER人工智能实验室）
Although federated learning has made awe-inspiring advances, most studies have assumed that the client's data are fully labeled. However, in a real-world scenario, every client may have a significant amount of unlabeled instances. Among the various approaches to utilizing unlabeled data, a federated active learning framework has emerged as a promising solution. In the decentralized setting, there are two types of available query selector models, namely 'global' and 'local-only' models, but little literature discusses their performance dominance and its causes. In this work, we first demonstrat…
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Guangzhou HKUST Fok Ying Tung Research Institute（广州香港科技大学霍英东研究院）；Baidu Research, Baidu Inc.（百度研究院，百度公司）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；BOSS Zhipin（BOSS直聘）
Task Diversity in Bayesian Federated Learning: Simultaneous Processing of Classification and Regression
School of Statistics, Renmin University of China（中国人民大学统计学院）；School of Statistics and Data Science, Southeast University（东南大学统计与数据科学学院）；Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing（未来区块链与隐私计算北京高精尖创新中心）
Southeast University（东南大学）；University of California, Berkeley（加州大学伯克利分校）；University of Michigan, Ann Arbor（密歇根大学安娜堡分校）；International Computer Science Institute（国际计算机科学研究所）
Federated learning (FL) is a promising way to use the computing power of mobile devices while maintaining the privacy of users. Current work in FL, however, makes the unrealistic assumption that the users have ground-truth labels on their devices, while also assuming that the server has neither data nor labels. In this work, we consider the more realistic scenario where the users have only unlabeled data, while the server has some labeled data, and where the amount of labeled data is smaller than the amount of unlabeled data. We call this learning problem semi-supervised federated learning (SS…
FedFlex: Federated Learning for Diverse Netflix Recommendations
Vrije Universiteit Amsterdam Netherlands；Georgia Institute of Technology Mexico；Centrum Wiskunde \& Informatica Netherlands；Georgia Institute of Technology
Australian Artificial Intelligence Institute (AAII)（澳大利亚人工智能研究所（AAII））
Open banking enables individual customers to own their banking data, which provides fundamental support for the boosting of a new ecosystem of data marketplaces and financial services. In the near future, it is foreseeable to have decentralized data ownership in the finance sector using federated learning. This is a just-in-time technology that can learn intelligent models in a decentralized training manner. The most attractive aspect of federated learning is its ability to decompose model training into a centralized server and distributed nodes without collecting private data. This kind of de…
Every Client Is an Environment: Federated De-confounding for Spatio-Temporal Forecasting
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Shanghai Jiao Tong University（上海交通大学）；Harbin Engineering University（哈尔滨工程大学）
In this manuscript, we propose a federated F-score based ensemble tree model for automatic rule extraction, namely Fed-FEARE. Under the premise of data privacy protection, Fed-FEARE enables multiple agencies to jointly extract set of rules both vertically and horizontally. Compared with that without federated learning, measures in evaluating model performance are highly improved. At present, Fed-FEARE has already been applied to multiple business, including anti-fraud and precision marketing, in a China nation-wide financial holdings group.
Tianjin University（天津大学）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）
Federated Learning (FL) is a collaborative machine learning technique to train a global model without obtaining clients' private data. The main challenges in FL are statistical diversity among clients, limited computing capability among clients' equipments, and the excessive communication overhead between the server and clients. To address these challenges, we propose a novel sparse personalized federated learning scheme via maximizing correlation (FedMac). By incorporating an approximated L1-norm and the correlation between client models and global model into standard FL loss function, the pe…
We show that a word-level recurrent neural network can predict emoji from text typed on a mobile keyboard. We demonstrate the usefulness of transfer learning for predicting emoji by pretraining the model using a language modeling task. We also propose mechanisms to trigger emoji and tune the diversity of candidates. The model is trained using a distributed on-device learning framework called federated learning. The federated model is shown to achieve better performance than a server-trained model. This work demonstrates the feasibility of using federated learning to train production-quality mo…
Microsoft Research（微软研究院）；Carnegie Mellon University（卡内基梅隆大学）
Federated learning (FL) enables edge-devices to collaboratively learn a model without disclosing their private data to a central aggregating server. Most existing FL algorithms require models of identical architecture to be deployed across the clients and server, making it infeasible to train large models due to clients' limited system resources. In this work, we propose a novel ensemble knowledge transfer method named Fed-ET in which small models (different in architecture) are trained on clients, and used to train a larger model at the server. Unlike in conventional ensemble learning, in FL …
Robust Federated Learning on Edge Devices with Domain Heterogeneity
Department of Computer Science and Engineering, Kyung Hee University, 171-04, Republic of Korea（韩国庆熙大学计算机科学与工程系）；Machine learning department, Mohamed Bin Zayed University of Artificial Intelligence, United Arab Emirates（阿联酋穆罕默德·本·扎耶德人工智能大学机器学习系）
The Diversity Bonus: Learning from Dissimilar Distributed Clients in Personalized Federated Learning
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；Jindal School of Management, The University of Texas at Dallas（德克萨斯大学达拉斯分校金德尔管理学院）；Department of Statistics, The University of Chicago（芝加哥大学统计系）；Department of Computing, The Hong Kong Polytechnic University（香港理工大学计算学系）
Machine learning abilities have become a vital component for various solutions across industries, applications, and sectors. Many organizations seek to leverage AI-based solutions across their business services to unlock better efficiency and increase productivity. Problems, however, can arise if there is a lack of quality data for AI-model training, scalability, and maintenance. We propose a data-centric federated learning architecture leveraged by a public blockchain and smart contracts to overcome this significant issue. Our proposed solution provides a virtual public marketplace where deve…
Federated learning (FL) is a new distributed machine learning framework that can achieve reliably collaborative training without collecting users' private data. However, due to FL's frequent communication and average aggregation strategy, they experience challenges scaling to statistical diversity data and large-scale models. In this paper, we propose a personalized FL framework, named Tensor Decomposition based Personalized Federated learning (TDPFed), in which we design a novel tensorized local model with tensorized linear layers and convolutional layers to reduce the communication cost. TDP…
Advances in Robust Federated Learning: A Survey with Heterogeneity Considerations
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；School of Software Engineering, Sun Yat-sen University（中山大学软件工程学院）；School of Systems Science and Engineering, Sun Yat-sen University（中山大学系统科学与工程学院）
Beijing University of Posts and Telecommunications（北京邮电大学）
Implementing existing federated learning in massive Internet of Things (IoT) networks faces critical challenges such as imbalanced and statistically heterogeneous data and device diversity. To this end, we propose a semi-federated learning (SemiFL) framework to provide a potential solution for the realization of intelligent IoT. By seamlessly integrating the centralized and federated paradigms, our SemiFL framework shows high scalability in terms of the number of IoT devices even in the presence of computing-limited sensors. Furthermore, compared to traditional learning approaches, the propose…
Federated Learning (FL) is a learning paradigm that protects privacy by keeping client data on edge devices. However, optimizing FL in practice can be difficult due to the diversity and heterogeneity of the learning system. Despite recent research efforts to improve the optimization of heterogeneous data, the impact of time-evolving heterogeneous data in real-world scenarios, such as changing client data or intermittent clients joining or leaving during training, has not been studied well. In this work, we propose Continual Federated Learning (CFL), a flexible framework for capturing the time-…
Xiamen University（厦门大学）；Hong Kong Baptist University（香港浸会大学）
Federated learning provides a privacy guarantee for generating good deep learning models on distributed clients with different kinds of data. Nevertheless, dealing with non-IID data is one of the most challenging problems for federated learning. Researchers have proposed a variety of methods to eliminate the negative influence of non-IIDness. However, they only focus on the non-IID data provided that the universal class distribution is balanced. In many real-world applications, the universal class distribution is long-tailed, which causes the model seriously biased. Therefore, this paper studi…
Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；Noah’s Ark Lab, Huawei（华为诺亚方舟实验室）
Federated learning faces huge challenges from model overfitting due to the lack of data and statistical diversity among clients. To address these challenges, this paper proposes a novel personalized federated learning method via Bayesian variational inference named pFedBayes. To alleviate the overfitting, weight uncertainty is introduced to neural networks for clients and the server. To achieve personalization, each client updates its local distribution parameters by balancing its construction error over private data and its KL divergence with global distribution from the server. Theoretical a…
Emulating Full Participation: An Effective and Fair Client Selection Strategy for Federated Learning
The Chinese University of Hong Kong（香港中文大学）；Zhejiang Lab（之江实验室）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区网络空间安全学院）；Tsinghua University（清华大学）
The Hong Kong Polytechnic University（香港理工大学）；Beijing Institute of Technology（北京理工大学）
Facing the challenge of statistical diversity in client local data distribution, personalized federated learning (PFL) has become a growing research hotspot. Although the state-of-the-art methods with model similarity-based pairwise collaboration have achieved promising performance, they neglect the fact that model aggregation is essentially a collaboration process within the coalition, where the complex multiwise influences take place among clients. In this paper, we first apply Shapley value (SV) from coalition game theory into the PFL scenario. To measure the multiwise collaboration among a…
