Federated learning, which solves the problem of data island by connecting multiple computational devices into a decentralized system, has become a promising paradigm for privacy-preserving machine learning. This paper studies vertical federated learning (VFL), which tackles the scenarios where collaborating organizations share the same set of users but disjoint features. Contemporary VFL methods are mainly used in static scenarios where the active party and the passive party have all the data from the beginning and will not change. However, the data in real life often changes dynamically. To alleviate this problem, we propose a new vertical federation learning method, DVFL, which adapts to dynamic data distribution changes through knowledge distillation. In DVFL, most of the computations are held locally to improve data security and model efficiency. Our extensive experimental results show that DVFL can not only obtain results close to existing VFL methods in static scenes, but also adapt to changes in data distribution in dynamic scenarios.
Recently, Graph Neural Network (GNN) has achieved remarkable success in various real-world problems on graph data. However in most industries, data exists in the form of isolated islands and the data privacy and security is also an important issue. In this paper, we propose FedVGCN, a federated GCN learning paradigm for privacy-preserving node classification task under data vertically partitioned setting, which can be generalized to existing GCN models. Specifically, we split the computation graph data into two parts. For each iteration of the training process, the two parties transfer interme…
Rensselaer Polytechnic Institute（伦斯勒理工学院）；IBM Research（IBM研究院）；National Yang Ming Chiao Tung University（国立阳明交通大学）
Recent studies show that private training data can be leaked through the gradients sharing mechanism deployed in distributed machine learning systems, such as federated learning (FL). Increasing batch size to complicate data recovery is often viewed as a promising defense strategy against data leakage. In this paper, we revisit this defense premise and propose an advanced data leakage attack with theoretical justification to efficiently recover batch data from the shared aggregated gradients. We name our proposed method as catastrophic data leakage in vertical federated learning (CAFE). Compar…
A Survey on Vertical Federated Learning: From a Layered Perspective
Hong Kong University of Science and Technology（香港科技大学）；Clustar Co., Ltd（Clustar有限公司）；Peking University（北京大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
PRIVEE: Privacy-Preserving Vertical Federated Learning Against Feature Inference Attacks
Department of Computer Science, Virginia Tech（维吉尼亚理工大学计算机科学系）；US DEVCOM Army Research Laboratory（美国陆军研发实验室）；Department of Computer Science, Old Dominion University（旧 Dominion 大学计算机科学系）；Department of Electrical and Computer Engineering, University of Arizona（亚利桑那大学电气与计算机工程系）
Vertical Federated Learning in Practice: The Good, the Bad, and the Ugly
National University of Singapore（新加坡国立大学）；WeBank（微众银行）；Zhejiang University（浙江大学）；Shanghai Jiao Tong University（上海交通大学）；Huazhong University of Science and Technology（华中科技大学）
HybridFL: A Federated Learning Approach for Financial Crime Detection
Department of Advanced Computing Sciences, Maastricht University（马斯特里赫特大学高级计算科学系）；Department of Cognitive Science and Artificial Intelligence, Tilburg University（蒂尔堡大学认知科学与人工智能系）
Vertical Federated Learning (vFL) allows multiple parties that own different attributes (e.g. features and labels) of the same data entity (e.g. a person) to jointly train a model. To prepare the training data, vFL needs to identify the common data entities shared by all parties. It is usually achieved by Private Set Intersection (PSI) which identifies the intersection of training samples from all parties by using personal identifiable information (e.g. email) as sample IDs to align data instances. As a result, PSI would make sample IDs of the intersection visible to all parties, and therefore…
Hijack Vertical Federated Learning Models As One Party
College of Computer Science and Technology at Zhejiang University（浙江大学计算机科学与技术学院）；School of Software Technology at Zhejiang University（浙江大学软件学院）；Hefei Interdisciplinary Center, National University of Defense Technology（国防科技大学合肥跨学科中心）；College of Information Science and Technology at Pennsylvania State University（宾夕法尼亚州立大学信息科学与技术学院）
Vertical Federated Learning (VFL) refers to the collaborative training of a model on a dataset where the features of the dataset are split among multiple data owners, while label information is owned by a single data owner. In this paper, we propose a novel method, Multi Vertical Federated Learning (Multi-VFL), to train VFL models when there are multiple data and label owners. Our approach is the first to consider the setting where $D$-data owners (across which features are distributed) and $K$-label owners (across which labels are distributed) exist. This proposed configuration allows differe…
Yale University（耶鲁大学）；Renmin University of China（中国人民大学）；JD Technology Group（京东科技集团）
Federated learning, conducive to solving data privacy and security problems, has attracted increasing attention recently. However, the existing federated boosting model sequentially builds a decision tree model with the weak base learner, resulting in redundant boosting steps and high interactive communication costs. In contrast, the federated bagging model saves time by building multi-decision trees in parallel, but it suffers from performance loss. With the aim of obtaining an outstanding performance with less time cost, we propose a novel model in a vertically federated setting termed as Fe…
More and more orgainizations and institutions make efforts on using external data to improve the performance of AI services. To address the data privacy and security concerns, federated learning has attracted increasing attention from both academia and industry to securely construct AI models across multiple isolated data providers. In this paper, we studied the efficiency problem of adapting widely used XGBoost model in real-world applications to vertical federated learning setting. State-of-the-art vertical federated XGBoost frameworks requires large number of encryption operations and ciphe…
Vertical federated learning is a collaborative machine learning framework to train deep leaning models on vertically partitioned data with privacy-preservation. It attracts much attention both from academia and industry. Unfortunately, applying most existing vertical federated learning methods in real-world applications still faces two daunting challenges. First, most existing vertical federated learning methods have a strong assumption that at least one party holds the complete set of labels of all data samples, while this assumption is not satisfied in many practical scenarios, where labels …
Federated learning is a distributed machine learning method that aims to preserve the privacy of sample features and labels. In a federated learning system, ID-based sample alignment approaches are usually applied with few efforts made on the protection of ID privacy. In real-life applications, however, the confidentiality of sample IDs, which are the strongest row identifiers, is also drawing much attention from many participants. To relax their privacy concerns about ID privacy, this paper formally proposes the notion of asymmetrical vertical federated learning and illustrates the way to pro…
A Survey of Privacy Threats and Defense in Vertical Federated Learning: From Model Life Cycle Perspective
Rensselaer Polytechnic Institute（伦斯勒理工学院）；Zhejiang University（浙江大学）；AntGroup（蚂蚁集团）；Ulsan National Institute of Science and Technology（蔚山国立科学技术院）；The University of Hong Kong（香港大学）
We present HDP-VFL, the first hybrid differentially private (DP) framework for vertical federated learning (VFL) to demonstrate that it is possible to jointly learn a generalized linear model (GLM) from vertically partitioned data with only a negligible cost, w.r.t. training time, accuracy, etc., comparing to idealized non-private VFL. Our work builds on the recent advances in VFL-based collaborative training among different organizations which rely on protocols like Homomorphic Encryption (HE) and Secure Multi-Party Computation (MPC) to secure computation and training. In particular, we analy…
JD Finance America Corporation（京东金融美国公司）；Department of Electrical & Computer Engineering, University of Pittsburgh（匹兹堡大学电气与计算机工程系）；University of Pittsburgh（匹兹堡大学）
The privacy-preserving federated learning for vertically partitioned data has shown promising results as the solution of the emerging multi-party joint modeling application, in which the data holders (such as government branches, private finance and e-business companies) collaborate throughout the learning process rather than relying on a trusted third party to hold data. However, existing federated learning algorithms for vertically partitioned data are limited to synchronous computation. To improve the efficiency when the unbalanced computation/communication resources are common among the pa…
Federated learning is a popular collaborative learning approach that enables clients to train a global model without sharing their local data. Vertical federated learning (VFL) deals with scenarios in which the data on clients have different feature spaces but share some overlapping samples. Existing VFL approaches suffer from high communication costs and cannot deal efficiently with limited overlapping samples commonly seen in the real world. We propose a practical vertical federated learning (VFL) framework called \textbf{one-shot VFL} that can solve the communication bottleneck and the prob…
WeBank（微众银行）；ShanghaiTech University（上海科技大学）；Hong Kong University of Science and Technology（香港科技大学）
Data privacy and security becomes a major concern in building machine learning models from different data providers. Federated learning shows promise by leaving data at providers locally and exchanging encrypted information. This paper studies the vertical federated learning structure for logistic regression where the data sets at two parties have the same sample IDs but own disjoint subsets of features. Existing frameworks adopt the first-order stochastic gradient descent algorithm, which requires large number of communication rounds. To address the communication challenge, we propose a quasi…
As data privacy is gradually valued by people, federated learning(FL) has emerged because of its potential to protect data. FL uses homomorphic encryption and differential privacy encryption on the promise of ensuring data security to realize distributed machine learning by exchanging encrypted information between different data providers. However, there are still many problems in FL, such as the communication efficiency between the client and the server and the data is non-iid. In order to solve the two problems mentioned above, we propose a novel vertical federated learning framework based o…
Institute for AI Industry Research, Tsinghua University（清华大学智能产业研究院）
Vertical Federated Learning (VFL) is widely utilized in real-world applications to enable collaborative learning while protecting data privacy and safety. However, previous works show that parties without labels (passive parties) in VFL can infer the sensitive label information owned by the party with labels (active party) or execute backdoor attacks to VFL. Meanwhile, active party can also infer sensitive feature information from passive party. All these pose new privacy and security challenges to VFL systems. We propose a new general defense method which limits the mutual information between…
University of Murcia（穆尔西亚大学）；University of Zurich（苏黎世大学）；armasuisse Science & Technology（瑞士联邦国防采购局科技部门）
Federated learning (FL) allows participants to collaboratively train machine and deep learning models while protecting data privacy. However, the FL paradigm still presents drawbacks affecting its trustworthiness since malicious participants could launch adversarial attacks against the training process. Related work has studied the robustness of horizontal FL scenarios under different attacks. However, there is a lack of work evaluating the robustness of decentralized vertical FL and comparing it with horizontal FL architectures affected by adversarial attacks. Thus, this work proposes three d…
As there is a growing interest in utilizing data across multiple resources to build better machine learning models, many vertically federated learning algorithms have been proposed to preserve the data privacy of the participating organizations. However, the efficiency of existing vertically federated learning algorithms remains to be a big problem, especially when applied to large-scale real-world datasets. In this paper, we present a fast, accurate, scalable and yet robust system for vertically federated random forest. With extensive optimization, we achieved $5\times$ and $83\times$ speed u…
Federated learning (FL) has recently emerged as a transformative paradigm that jointly train a model with distributed data sets in IoT while avoiding the need for central data collection. Due to the limited observation range, such data sets can only reflect local information, which limits the quality of trained models. In practice, the global information and local observations would require a joint consideration for learning to make a reasonable policy. However, in horizontal FL, the central agency only acts as a model aggregator without utilizing its global observation to further improve the …
Eliminating Label Leakage in Tree-Based Vertical Federated Learning
The University of Tokyo（东京大学）；Institute for AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Federated Learning is a new distributed learning mechanism which allows model training on a large corpus of decentralized data owned by different data providers, without sharing or leakage of raw data. According to the characteristics of data dis-tribution, it could be usually classified into three categories: horizontal federated learning, vertical federated learning, and federated transfer learning. In this paper we present a solution for parallel dis-tributed logistic regression for vertical federated learning. As compared with existing works, the role of third-party coordinator is removed …
School of Computer Science and Engineering, Nanyang Technological University, Singapore（南洋理工大学计算机科学与工程学院，新加坡）；Joint NTU-WeBank Research Centre on FinTech, NTU, Singapore（南洋理工大学-微众银行金融科技联合研究中心，新加坡南洋理工大学）
Federated learning (FL) is a privacy-preserving paradigm for training collective machine learning models with locally stored data from multiple participants. Vertical federated learning (VFL) deals with the case where participants sharing the same sample ID space but having different feature spaces, while label information is owned by one participant. Current studies of VFL only support two participants, and mostly focus on binaryclass logistic regression problems. In this paper, we propose the Multi-participant Multi-class Vertical Federated Learning (MMVFL) framework for multi-class VFL prob…
Vertical federated learning (VFL) allows an active party with labeled feature to leverage auxiliary features from the passive parties to improve model performance. Concerns about the private feature and label leakage in both the training and inference phases of VFL have drawn wide research attention. In this paper, we propose a general privacy-preserving vertical federated deep learning framework called FedPass, which leverages adaptive obfuscation to protect the feature and label simultaneously. Strong privacy-preserving capabilities about private features and labels are theoretically proved …
Federated learning has gained great attention recently as a privacy-enhancing tool to jointly train a machine learning model by multiple parties. As a sub-category, vertical federated learning (vFL) focuses on the scenario where features and labels are split into different parties. The prior work on vFL has mostly studied how to protect label privacy during model training. However, model evaluation in vFL might also lead to potential leakage of private label information. One mitigation strategy is to apply label differential privacy (DP) but it gives bad estimations of the true (non-private) m…
The Hong Kong University of Science and Technology（香港科技大学）；Clustar（Clustar（星环聚能））；Peking University（北京大学）；Fuzhou University（福州大学）
Vertical federated learning (VFL) is attracting much attention because it enables cross-silo data cooperation in a privacy-preserving manner. While most research works in VFL focus on linear and tree models, deep models (e.g., neural networks) are not well studied in VFL. In this paper, we focus on SplitNN, a well-known neural network framework in VFL, and identify a trade-off between data security and model performance in SplitNN. Briefly, SplitNN trains the model by exchanging gradients and transformed data. On the one hand, SplitNN suffers from the loss of model performance since multiply p…
iSING Lab, Hong Kong University of Science and Technology（香港科技大学iSING实验室）；Clustar Technology Co., Ltd.（星云科技股份有限公司）；Peng Cheng Lab（鹏城实验室）
Vertical federated learning (VFL) leverages various privacy-preserving algorithms, e.g., homomorphic encryption or secret sharing based SecureBoost, to ensure data privacy. However, these algorithms all require a semi-honest secure definition, which raises concerns in real-world applications. In this paper, we present Aegis, a trusted, automatic, and accurate verification framework to verify the security of VFL jobs. Aegis is separated from local parties to ensure the security of the framework. Furthermore, it automatically adapts to evolving VFL algorithms by defining the VFL job as a finite …
As more and more companies store their customers' data; various information of a person is distributed among numerous companies' databases. Different industrial sectors carry distinct features about the same customers. Also, different companies within the same industrial sector carry similar kinds of data about the customers with different data representations. Cooperation between companies from different industrial sectors, called vertical cooperation, and between the companies within the same sector, called horizontal cooperation, can lead to more accurate machine learning models and better …
Vertical Federated Learning: Concepts, Advances and Challenges
Institute for AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）；Webank（微众银行）；AsiaInfo Technologies（亚信科技）；Hong Kong University of Science and Technology（香港科技大学）
OpenMined；Edinburgh Napier University（爱丁堡龙比亚大学）；Apheris；Tessella；Birla Institute of Technology, Mesra（比拉理工学院（梅斯拉校区））
We introduce PyVertical, a framework supporting vertical federated learning using split neural networks. The proposed framework allows a data scientist to train neural networks on data features vertically partitioned across multiple owners while keeping raw data on an owner's device. To link entities shared across different datasets' partitions, we use Private Set Intersection on IDs associated with data points. To demonstrate the validity of the proposed framework, we present the training of a simple dual-headed split neural network for a MNIST classification task, with data samples verticall…
Xidian University（西安电子科技大学）；JD Tech（京东科技）；JD Finance America Corporation（京东金融美国公司）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；University of Pittsburgh（匹兹堡大学）
Vertical federated learning (VFL) attracts increasing attention due to the emerging demands of multi-party collaborative modeling and concerns of privacy leakage. In the real VFL applications, usually only one or partial parties hold labels, which makes it challenging for all parties to collaboratively learn the model without privacy leakage. Meanwhile, most existing VFL algorithms are trapped in the synchronous computations, which leads to inefficiency in their real-world applications. To address these challenging problems, we propose a novel {\bf VF}L framework integrated with new {\bf b}ack…
IBM Research（IBM研究院）；University of Pittsburgh（匹兹堡大学）
Federated learning (FL) has been proposed to allow collaborative training of machine learning (ML) models among multiple parties where each party can keep its data private. In this paradigm, only model updates, such as model weights or gradients, are shared. Many existing approaches have focused on horizontal FL, where each party has the entire feature set and labels in the training data set. However, many real scenarios follow a vertically-partitioned FL setup, where a complete feature set is formed only when all the datasets from the parties are combined, and the labels are only available to…
