We consider learning a multi-class classification model in the federated setting, where each user has access to the positive data associated with only a single class. As a result, during each federated learning round, the users need to locally update the classifier without having access to the features and the model parameters for the negative classes. Thus, naively employing conventional decentralized learning such as the distributed SGD or Federated Averaging may lead to trivial or extremely poor classifiers. In particular, for the embedding based classifiers, all the class embeddings might collapse to a single point.
To address this problem, we propose a generic framework for training with only positive labels, namely Federated Averaging with Spreadout (FedAwS), where the server imposes a geometric regularizer after each round to encourage classes to be spreadout in the embedding space. We show, both theoretically and empirically, that FedAwS can almost match the performance of conventional learning where users have access to negative labels. We further extend the proposed method to the settings with large output spaces.
While existing federated learning approaches mostly require that clients have fully-labeled data to train on, in realistic settings, data obtained at the client-side often comes without any accompanying labels. Such deficiency of labels may result from either high labeling cost, or difficulty of annotation due to the requirement of expert knowledge. Thus the private data at each client may be either partly labeled, or completely unlabeled with labeled data being available only at the server, which leads us to a new practical federated learning problem, namely Federated Semi-Supervised Learning…
Vertical federated learning is a collaborative machine learning framework to train deep leaning models on vertically partitioned data with privacy-preservation. It attracts much attention both from academia and industry. Unfortunately, applying most existing vertical federated learning methods in real-world applications still faces two daunting challenges. First, most existing vertical federated learning methods have a strong assumption that at least one party holds the complete set of labels of all data samples, while this assumption is not satisfied in many practical scenarios, where labels …
Federated Learning with Heterogeneous and Private Label Sets
Chalmers University of Technology（查尔姆斯理工大学）；RISE Research Institutes of Sweden（瑞典RISE研究院）；University of Gothenburg（哥德堡大学）；KTH Royal Institute of Technology（瑞典皇家理工学院）
Industrial Technology Research Institute（工业技术研究院）；National Yang Ming Chiao Tung University（阳明交通大学）
Federated learning aims to share private data to maximize the data utility without privacy leakage. Previous federated learning research mainly focuses on multi-class classification problems. However, multi-label classification is a crucial research problem close to real-world data properties. Nevertheless, a limited number of federated learning studies explore this research problem. Existing studies of multi-label federated learning did not consider the characteristics of multi-label data, i.e., they used the concept of multi-class classification to verify their methods' performance, which me…
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Nanyang Technological University（南洋理工大学）；Xuanwu Hospital, Capital Medical University（首都医科大学宣武医院）
Ubiquitous systems with End-Edge-Cloud architecture are increasingly being used in healthcare applications. Federated Learning (FL) is highly useful for such applications, due to silo effect and privacy preserving. Existing FL approaches generally do not account for disparities in the quality of local data labels. However, the clients in ubiquitous systems tend to suffer from label noise due to varying skill-levels, biases or malicious tampering of the annotators. In this paper, we propose Federated Opportunistic Computing for Ubiquitous Systems (FOCUS) to address this challenge. It maintains …
Federated learning enables many local devices to train a deep learning model jointly without sharing the local data. Currently, most of federated training schemes learns a global model by averaging the parameters of local models. However, most of these training schemes suffer from high communication cost resulted from transmitting full local model parameters. Moreover, directly averaging model parameters leads to a significant performance degradation, due to the class-imbalanced non-iid data on different devices. Especially for the real life federated learning tasks involving extreme classific…
Vertical Federated Learning (VFL) refers to the collaborative training of a model on a dataset where the features of the dataset are split among multiple data owners, while label information is owned by a single data owner. In this paper, we propose a novel method, Multi Vertical Federated Learning (Multi-VFL), to train VFL models when there are multiple data and label owners. Our approach is the first to consider the setting where $D$-data owners (across which features are distributed) and $K$-label owners (across which labels are distributed) exist. This proposed configuration allows differe…
School of Computer Science and Engineering, Nanyang Technological University, Singapore（南洋理工大学计算机科学与工程学院，新加坡）；Joint NTU-WeBank Research Centre on FinTech, NTU, Singapore（南洋理工大学-微众银行金融科技联合研究中心，新加坡南洋理工大学）
Federated learning (FL) is a privacy-preserving paradigm for training collective machine learning models with locally stored data from multiple participants. Vertical federated learning (VFL) deals with the case where participants sharing the same sample ID space but having different feature spaces, while label information is owned by one participant. Current studies of VFL only support two participants, and mostly focus on binaryclass logistic regression problems. In this paper, we propose the Multi-participant Multi-class Vertical Federated Learning (MMVFL) framework for multi-class VFL prob…
Label-shift robust federated feature screening for high-dimensional classification
Center for Applied Statistics and School of Statistics Renmin University of China（应用统计中心和中国人民大学统计学院）；School of Mathematics and Statistics Xi’an Jiaotong University（西安交通大学数学与统计学院）；Department of Mathematics and Fundamental Research Peng Cheng Laboratory School of Mathematics and Statistics Xi’an Jiaotong University（数学与基础研究部鹏城实验室数学与统计学院）
Little is Enough: Boosting Privacy by Sharing Only Hard Labels in Federated Semi-Supervised Learning
Institute for Neuroinformatics, Ruhr University Bochum（波鸿鲁尔大学神经信息学研究所）；Monash University（莫纳什大学）；Case Western Reserve University（凯斯西储大学）；Carenostics（Carenostics公司）
University of Pittsburgh（匹兹堡大学）；Simon Fraser University（西蒙弗雷泽大学）
Federated learning (FL) is a promising privacy-preserving machine learning paradigm over distributed located data. In FL, the data is kept locally by each user. This protects the user privacy, but also makes the server difficult to verify data quality, especially if the data are correctly labeled. Training with corrupted labels is harmful to the federated learning task; however, little attention has been paid to FL in the case of label noise. In this paper, we focus on this problem and propose a learning-based reweighting approach to mitigate the effect of noisy labels in FL. More precisely, w…
CATCHFed: Efficient Unlabeled Data Utilization for Semi-Supervised Federated Learning in Limited Labels Environments
Dept. of Computer and Information Science, Korea University, Sejong, South Korea（韩国大学计算机与信息科学系）；Department of Computer Science, University of Surrey, Guildford, United Kingdom（塞弗里大学计算机科学系）；Dept. of Semiconductor Engineering, Seoul National University of Science and Technology, Seoul, South Korea（首尔科学技术大学半导体工程系）；Dept. of Electrical and Information Engineering, Seoul National University of Science and Technology, Seoul, South Korea（首尔科学技术大学电气与信息工程系）
Label Granularity Skew in Federated Learning with Hierarchical Image Classification
School of Computer Science and Engineering, Soongsil University（崇实大学计算机科学与工程学院）；School of Computing and Augmented Intelligence, Arizona State University（亚利桑那州立大学计算与增强智能学院）
Pennsylvania State University（宾夕法尼亚州立大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Purdue University（普渡大学）；Georgia State University（佐治亚州立大学）
Federated Learning has shown great potentials for the distributed data utilization and privacy protection. Most existing federated learning approaches focus on the supervised setting, which means all the data stored in each client has labels. However, in real-world applications, the client data are impossible to be fully labeled. Thus, how to exploit the unlabeled data should be a new challenge for federated learning. Although a few studies are attempting to overcome this challenge, they may suffer from information leakage or misleading information usage problems. To tackle these issues, in th…
Central South University（中南大学）；University of South Florida（南佛罗里达大学）
Federated learning achieves joint training of deep models by connecting decentralized data sources, which can significantly mitigate the risk of privacy leakage. However, in a more general case, the distributions of labels among clients are different, called ``label distribution skew''. Directly applying conventional federated learning without consideration of label distribution skew issue significantly hurts the performance of the global model. To this end, we propose a novel federated learning method, named FedMGD, to alleviate the performance degradation caused by the label distribution ske…
Data privacy has become an increasingly important concern in real-world big data applications such as machine learning. To address the problem, federated learning (FL) has been a promising solution to building effective machine learning models from decentralized and private data. Existing federated learning algorithms mainly tackle the supervised learning problem, where data are assumed to be fully labeled. However, in practice, fully labeled data is often hard to obtain, as the participants may not have sufficient domain expertise, or they lack the motivation and tools to label data. Therefor…
In real-world federated learning scenarios, participants could have their own personalized labels which are incompatible with those from other clients, due to using different label permutations or tackling completely different tasks or domains. However, most existing FL approaches cannot effectively tackle such extremely heterogeneous scenarios since they often assume that (1) all participants use a synchronized set of labels, and (2) they train on the same task from the same domain. In this work, to tackle these challenges, we introduce Factorized-FL, which allows to effectively tackle label-…
Federated learning is a paradigm that enables local devices to jointly train a server model while keeping the data decentralized and private. In federated learning, since local data are collected by clients, it is hardly guaranteed that the data are correctly annotated. Although a lot of studies have been conducted to train the networks robust to these noisy data in a centralized setting, these algorithms still suffer from noisy labels in federated learning. Compared to the centralized setting, clients' data can have different noise distributions due to variations in their labeling systems or …
Machine learning in medical research, by nature, needs careful attention on obeying the regulations of data privacy, making it difficult to train a machine learning model over gathered data from different medical centers. Failure of leveraging data of the same kind may result in poor generalizability for the trained model. Federated learning (FL) enables collaboratively training a joint model while keeping the data decentralized for multiple medical centers. However, federated optimizations often suffer from the heterogeneity of the data distribution across medical centers. In this work, we pr…
Global AI Accelerator, Ericsson（爱立信全球人工智能加速器）；Ericsson Research（爱立信研究院）
Various health-care applications such as assisted living, fall detection, etc., require modeling of user behavior through Human Activity Recognition (HAR). Such applications demand characterization of insights from multiple resource-constrained user devices using machine learning techniques for effective personalized activity monitoring. On-device Federated Learning proves to be an effective approach for distributed and collaborative machine learning. However, there are a variety of challenges in addressing statistical (non-IID data) and model heterogeneities across users. In addition, in this…
University of Southern California（南加州大学）；Emory University（埃默里大学）；Xidian University（西安电子科技大学）
Federated learning enables multiple clients, such as mobile phones and organizations, to collaboratively learn a shared model for prediction while protecting local data privacy. However, most recent research and applications of federated learning assume that all clients have fully labeled data, which is impractical in real-world settings. In this work, we focus on a new scenario for cross-silo federated learning, where data samples of each client are partially labeled. We borrow ideas from semi-supervised learning methods where a large amount of unlabeled data is utilized to improve the model'…
FedNoRo: Towards Noise-Robust Federated Learning by Addressing Class Imbalance and Label Noise Heterogeneity
Huazhong University of Science and Technology（华中科技大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Hong Kong University of Science and Technology（香港科技大学）
FedReLa: Imbalanced Federated Learning via Re-Labeling
School of Computing and Information Systems, University of Melbourne, Victoria, Australia（墨尔本大学计算机与信息系统学院）；School of Statistics and Data Science, LPMC, KLMDASR, and LEBPS, Nankai University（南开大学统计与数据科学学院）；School of Mathematics and Statistics, University of Melbourne, Victoria, Australia（墨尔本大学数学与统计学学院）
Federated Unlearning Model Recovery in Data with Skewed Label Distributions
School of Computer Science and Technology（计算机科学与技术学院）；Dalian University of Technology（大连理工大学）；School of Engineering and Computer Science（工程与计算机科学学院）；Victoria University of Wellington（惠灵顿维多利亚大学）
Overcoming label shift with target-aware federated learning
RISE Research Institutes of Sweden（瑞典RISE研究机构）；KTH Royal Institute of Technology（皇家理工学院）；Chalmers University of Technology（查尔姆斯理工大学）；University of Gothenburg（哥德堡大学）
Institute for AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）；Tsinghua University（清华大学）；Webank（微众银行）；Shandong University（山东大学）；Hong Kong University of Science and Technology（香港科技大学）
In a vertical federated learning (VFL) scenario where features and model are split into different parties, communications of sample-specific updates are required for correct gradient calculations but can be used to deduce important sample-level label information. An immediate defense strategy is to protect sample-level messages communicated with Homomorphic Encryption (HE), and in this way only the batch-averaged local gradients are exposed to each party (termed black-boxed VFL). In this paper, we first explore the possibility of recovering labels in the vertical federated learning setting wit…
Global AI Accelerator, Ericsson（爱立信全球人工智能加速器）；Indian Institute of Technology, Madras（马德拉斯印度理工学院）；Ericsson Research（爱立信研究院）
Various IoT applications demand resource-constrained machine learning mechanisms for different applications such as pervasive healthcare, activity monitoring, speech recognition, real-time computer vision, etc. This necessitates us to leverage information from multiple devices with few communication overheads. Federated Learning proves to be an extremely viable option for distributed and collaborative machine learning. Particularly, on-device federated learning is an active area of research, however, there are a variety of challenges in addressing statistical (non-IID data) and model heterogen…
University of Maryland（马里兰大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Numerous large-scale chest x-ray datasets have spearheaded expert-level detection of abnormalities using deep learning. However, these datasets focus on detecting a subset of disease labels that could be present, thus making them distributed and non-iid with partial labels. Recent literature has indicated the impact of batch normalization layers on the convergence of federated learning due to domain shift associated with non-iid data with partial labels. To that end, we propose FedFBN, a federated learning framework that draws inspiration from transfer learning by using pretrained networks as …
Federated learning has gained great attention recently as a privacy-enhancing tool to jointly train a machine learning model by multiple parties. As a sub-category, vertical federated learning (vFL) focuses on the scenario where features and labels are split into different parties. The prior work on vFL has mostly studied how to protect label privacy during model training. However, model evaluation in vFL might also lead to potential leakage of private label information. One mitigation strategy is to apply label differential privacy (DP) but it gives bad estimations of the true (non-private) m…
Institute for AI Industry Research, Tsinghua University（清华大学智能产业研究院）
Vertical Federated Learning (VFL) is widely utilized in real-world applications to enable collaborative learning while protecting data privacy and safety. However, previous works show that parties without labels (passive parties) in VFL can infer the sensitive label information owned by the party with labels (active party) or execute backdoor attacks to VFL. Meanwhile, active party can also infer sensitive feature information from passive party. All these pose new privacy and security challenges to VFL systems. We propose a new general defense method which limits the mutual information between…
Australian Artificial Intelligence Institute, University of Technology Sydney（悉尼科技大学澳大利亚人工智能研究所）；University of Washington, Seattle（西雅图华盛顿大学）；University of Maryland, College Park（马里兰大学帕克分校）；Hong Kong Baptist University（香港浸会大学）
Federated learning (FL) aims at training a global model on the server side while the training data are collected and located at the local devices. Hence, the labels in practice are usually annotated by clients of varying expertise or criteria and thus contain different amounts of noises. Local training on noisy labels can easily result in overfitting to noisy labels, which is devastating to the global model through aggregation. Although recent robust FL methods take malicious clients into account, they have not addressed local noisy labels on each device and the impact to the global model. In …
Duke University（杜克大学）；University of Minnesota-Twin Cities（明尼苏达大学双城分校）
Federated Learning allows the training of machine learning models by using the computation and private data resources of many distributed clients. Most existing results on Federated Learning (FL) assume the clients have ground-truth labels. However, in many practical scenarios, clients may be unable to label task-specific data due to a lack of expertise or resource. We propose SemiFL to address the problem of combining communication-efficient FL such as FedAvg with Semi-Supervised Learning (SSL). In SemiFL, clients have completely unlabeled data and can train multiple local epochs to reduce co…
Learning Locally, Revising Globally: Global Reviser for Federated Learning with Noisy Labels
College of Computer Science, Sichuan University, Chengdu, China（四川大学计算机学院，中国成都）；Engineering Research Center of Machine Learning（机器学习工程研究中心）；University of Sydney, Sydney, Australia（悉尼大学，澳大利亚悉尼）；Southeast University, Nanjing, China（东南大学，中国南京）
Federated learning (FL) has recently emerged as a new form of collaborative machine learning, where a common model can be learned while keeping all the training data on local devices. Although it is designed for enhancing the data privacy, we demonstrated in this paper a new direction in inference attacks in the context of FL, where valuable information about training data can be obtained by adversaries with very limited power. In particular, we proposed three new types of attacks to exploit this vulnerability. The first type of attack, Class Sniffing, can detect whether a certain label appear…
FedFusion: Federated Learning with Diversity- and Cluster-Aware Encoders for Robust Adaptation under Label Scarcity
Department of Computer Science University of the Western Cape（计算机科学系乌得勒支大学）；Department of Computer Science Missouri University of Science and Technology（计算机科学系密苏里科技大学）
Traditional federated optimization methods perform poorly with heterogeneous data (ie, accuracy reduction), especially for highly skewed data. In this paper, we investigate the label distribution skew in FL, where the distribution of labels varies across clients. First, we investigate the label distribution skew from a statistical view. We demonstrate both theoretically and empirically that previous methods based on softmax cross-entropy are not suitable, which can result in local models heavily overfitting to minority classes and missing classes. Additionally, we theoretically introduce a dev…
We study the problem of learning from positive and unlabeled (PU) data in the federated setting, where each client only labels a little part of their dataset due to the limitation of resources and time. Different from the settings in traditional PU learning where the negative class consists of a single class, the negative samples which cannot be identified by a client in the federated setting may come from multiple classes which are unknown to the client. Therefore, existing PU learning methods can be hardly applied in this situation. To address this problem, we propose a novel framework, name…
Warsaw University of Technology（华沙理工大学）；Systems Research Institute Polish Academy of Sciences（波兰科学院系统研究所）
One of the important problems in federated learning is how to deal with unbalanced data. This contribution introduces a novel technique designed to deal with label skewed non-IID data, using adversarial inputs, created by the I-FGSM method. Adversarial inputs guide the training process and allow the Weighted Federated Averaging to give more importance to clients with 'selected' local label distributions. Experimental results, gathered from image classification tasks, for MNIST and CIFAR-10 datasets, are reported and analyzed.
Vertical federated learning (vFL) has gained much attention and been deployed to solve machine learning problems with data privacy concerns in recent years. However, some recent work demonstrated that vFL is vulnerable to privacy leakage even though only the forward intermediate embedding (rather than raw features) and backpropagated gradients (rather than raw labels) are communicated between the involved participants. As the raw labels often contain highly sensitive information, some recent work has been proposed to prevent the label leakage from the backpropagated gradients effectively in vF…
Federated learning enables multiple users to build a joint model by sharing their model updates (gradients), while their raw data remains local on their devices. In contrast to the common belief that this provides privacy benefits, we here add to the very recent results on privacy risks when sharing gradients. Specifically, we investigate Label Leakage from Gradients (LLG), a novel attack to extract the labels of the users' training data from their shared gradients. The attack exploits the direction and magnitude of gradients to determine the presence or absence of any label. LLG is simple yet…
Fraunhofer Heinrich Hertz Institute（弗劳恩霍夫海因里希·赫兹研究所）
Communication constraints are one of the major challenges preventing the wide-spread adoption of Federated Learning systems. Recently, Federated Distillation (FD), a new algorithmic paradigm for Federated Learning with fundamentally different communication properties, emerged. FD methods leverage ensemble distillation techniques and exchange model outputs, presented as soft labels on an unlabeled public data set, between the central server and the participating clients. While for conventional Federated Learning algorithms, like Federated Averaging (FA), communication scales with the size of th…
Vertical federated learning (VFL) allows an active party with labeled feature to leverage auxiliary features from the passive parties to improve model performance. Concerns about the private feature and label leakage in both the training and inference phases of VFL have drawn wide research attention. In this paper, we propose a general privacy-preserving vertical federated deep learning framework called FedPass, which leverages adaptive obfuscation to protect the feature and label simultaneously. Strong privacy-preserving capabilities about private features and labels are theoretically proved …
Federated Learning with Label-Masking Distillation
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, UCAS（中国科学院大学网络空间安全学院）；School of Computer Science, Fudan University（复旦大学计算机科学技术学院）
