Federated Learning (FL) and Split Learning (SL) are two popular paradigms of distributed machine learning. By offloading the computation-intensive portions to the server, SL is promising for deep model training on resource-constrained devices, yet still lacking of rigorous convergence analysis. In this paper, we derive the convergence guarantees of Sequential SL (SSL, the vanilla case of SL that conducts the model training in sequence) for strongly/general/non-convex objectives on heterogeneous data. Notably, the derived guarantees suggest that SSL is better than Federated Averaging (FedAvg, the most popular algorithm in FL) on heterogeneous data. We validate the counterintuitive analysis result empirically on extremely heterogeneous data.
Convergence Analysis of Split Federated Learning on Heterogeneous Data
Guangdong University of Technology（广东工业大学）；Montclair State University（蒙特克莱尔州立大学）；Southern University of Science and Technology（南方科技大学）；University of California, Davis（加利福尼亚大学戴维斯分校）
University of Edinburgh（爱丁堡大学）；AndPlus LLC（AndPlus有限责任公司）
Neural networks have been able to achieve groundbreaking accuracy at tasks conventionally considered only doable by humans. Using stochastic gradient descent, optimization in many dimensions is made possible, albeit at a relatively high computational cost. By splitting training data into batches, networks can be distributed and trained vastly more efficiently and with minimal accuracy loss. We have explored the mathematics behind efficiently implementing tensor-based batch backpropagation algorithms. A common approach to batch training is iterating over batch items individually. Explicitly usi…
Seq-HGNN: Learning Sequential Node Representation on Heterogeneous Graph
Beihang University（北京航空航天大学）；Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；BOSS Zhipin（BOSS直聘）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Federated Learning (FL) and Split Learning (SL) are privacy-preserving Machine-Learning (ML) techniques that enable training ML models over data distributed among clients without requiring direct access to their raw data. Existing FL and SL approaches work on horizontally or vertically partitioned data and cannot handle sequentially partitioned data where segments of multiple-segment sequential data are distributed across clients. In this paper, we propose a novel federated split learning framework, FedSL, to train models on distributed sequential data. The most common ML models to train on se…
Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders
AIRI, Skoltech Moscow Russian Federation（AIRI，斯克里普钦斯基因工学院莫斯科俄罗斯联邦）；Sber AI Lab, Skoltech Moscow Russian Federation（Sber AI Lab，斯克里普钦斯基因工学院莫斯科俄罗斯联邦）；Sber AI Lab Moscow Russian Federation（Sber AI Lab，莫斯科斯克里普钦斯基因工学院俄罗斯联邦）；Sber AI Lab, HSE University Moscow Russian Federation（Sber AI Lab，莫斯科高等经济大学俄罗斯联邦）
Generating Tabular Data Using Heterogeneous Sequential Feature Forest Flow Matching
AIMS Research and Innovation Center（AIMS研究与创新中心）；Mila Québec AI Institute（魁北克AI研究所）；Université de Montréal（蒙特利尔大学）；SAIT Montreal AI Lab（蒙特利尔SAIT人工智能实验室）
With the frequent happening of privacy leakage and the enactment of privacy laws across different countries, data owners are reluctant to directly share their raw data and labels with any other party. In reality, a lot of these raw data are stored in the graph database, especially for finance. For collaboratively building graph neural networks(GNNs), federated learning(FL) may not be an ideal choice for the vertically partitioned setting where privacy and efficiency are the main concerns. Moreover, almost all the existing federated GNNs are mainly designed for homogeneous graphs, which simplif…
University of Michigan（密歇根大学）；Indian Institute of Technology Indore（印度理工学院印多尔分校）；University of Manitoba（曼尼托巴大学）；National Taipei University of Technology（台北科技大学）；Qatar University（卡塔尔大学）
Ant Group（蚂蚁集团）；Tianjin University（天津大学）；Shantou University（汕头大学）
Sequential recommendation (SR) learns users' preferences by capturing the sequential patterns from users' behaviors evolution. As discussed in many works, user-item interactions of SR generally present the intrinsic power-law distribution, which can be ascended to hierarchy-like structures. Previous methods usually handle such hierarchical information by making user-item sectionalization empirically under Euclidean space, which may cause distortion of user-item representation in real online scenarios. In this paper, we propose a Poincaré-based heterogeneous graph neural network named PHGR to m…
GAPSL: A Gradient-Aligned Parallel Split Learning over Data-Heterogeneous Edge Computing Systems
Department of Electrical and Electronic Engineering, University of Hong Kong（香港大学电子与电气工程系）；Data61, CSIRO（CSIRO数据61）；Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg（卢森堡大学安全、可靠与信任跨学科中心）
Virginia Polytechnic Institute and State University（弗吉尼亚理工学院暨州立大学）；U.S. Army Research Laboratory（美国陆军研究实验室）
Gaussian processes (GPs) are a well-known nonparametric Bayesian inference technique, but they suffer from scalability problems for large sample sizes, and their performance can degrade for non-stationary or spatially heterogeneous data. In this work, we seek to overcome these issues through (i) employing variational free energy approximations of GPs operating in tandem with online expectation propagation steps; and (ii) introducing a local splitting step which instantiates a new GP whenever the posterior distribution changes significantly as quantified by the Wasserstein metric over posterior…
Politecnico di Milano（米兰理工大学）；Technische Universität Dresden（德累斯顿工业大学）
Survival analysis studies time-modeling techniques for an event of interest occurring for a population. Survival analysis found widespread applications in healthcare, engineering, and social sciences. However, the data needed to train survival models are often distributed, incomplete, censored, and confidential. In this context, federated learning can be exploited to tremendously improve the quality of the models trained on distributed data while preserving user privacy. However, federated survival analysis is still in its early development, and there is no common benchmarking dataset to test …
Guangdong University of Technology（广东工业大学）；Tencent Technology (SZ) Co., Ltd.（腾讯科技（深圳）有限公司）；Cornell University（康奈尔大学）
Sequential recommendation aims to choose the most suitable items for a user at a specific timestamp given historical behaviors. Existing methods usually model the user behavior sequence based on the transition-based methods like Markov Chain. However, these methods also implicitly assume that the users are independent of each other without considering the influence between users. In fact, this influence plays an important role in sequence recommendation since the behavior of a user is easily affected by others. Therefore, it is desirable to aggregate both user behaviors and the influence betwe…
Kyoto University（京都大学）；RIKEN AIP（理化学研究所先进智能项目中心）；NTT Communication Science Laboratories（NTT通信科学实验室）
In decentralized learning, operator splitting methods using a primal-dual formulation (e.g., the Edge-Consensus Learning (ECL)) has been shown to be robust to heterogeneous data and has attracted significant attention in recent years. However, in the ECL, a node needs to exchange dual variables with its neighbors. These exchanges incur significant communication costs. For the Gossip-based algorithms, many compression methods have been proposed, but these Gossip-based algorithm do not perform well when the data distribution held by each node is statistically heterogeneous. In this work, we prop…
Federated Learning (FL) allows training machine learning models in privacy-constrained scenarios by enabling the cooperation of edge devices without requiring local data sharing. This approach raises several challenges due to the different statistical distribution of the local datasets and the clients' computational heterogeneity. In particular, the presence of highly non-i.i.d. data severely impairs both the performance of the trained neural network and its convergence rate, increasing the number of communication rounds requested to reach a performance comparable to that of the centralized sc…
Georgia Institute of Technology（佐治亚理工学院）；Microsoft Research India（微软印度研究院）；Microsoft（微软）；Microsoft Research AI（微软人工智能研究院）
For many kinds of interventions, such as a new advertisement, marketing intervention, or feature recommendation, it is important to target a specific subset of people for maximizing its benefits at minimum cost or potential harm. However, a key challenge is that no data is available about the effect of such a prospective intervention since it has not been deployed yet. In this work, we propose a split-treatment analysis that ranks the individuals most likely to be positively affected by a prospective intervention using past observational data. Unlike standard causal inference methods, the spli…
Massachusetts Institute of Technology（麻省理工学院）；University of British Columbia（不列颠哥伦比亚大学）；Sharif University of Technology（谢里夫理工大学）
Physics-informed neural networks (PINNs) have received significant attention as a unified framework for forward, inverse, and surrogate modeling of problems governed by partial differential equations (PDEs). Training PINNs for forward problems, however, pose significant challenges, mainly because of the complex non-convex and multi-objective loss function. In this work, we present a PINN approach to solving the equations of coupled flow and deformation in porous media for both single-phase and multiphase flow. To this end, we construct the solution space using multi-layer neural networks. Due …
ENNEW Digital Research Institute, ENN Group（新奥集团新智数字研究院）；Institute for Infocomm Research, A*STAR（新加坡科技研究局资讯通信研究院）；National University of Singapore（新加坡国立大学）
Federated learning (FL) and split learning (SL) are the two popular distributed machine learning (ML) approaches that provide some data privacy protection mechanisms. In the time-series classification problem, many researchers typically use 1D convolutional neural networks (1DCNNs) based on the SL approach with a single client to reduce the computational overhead at the client-side while still preserving data privacy. Another method, recurrent neural network (RNN), is utilized on sequentially partitioned data where segments of multiple-segment sequential data are distributed across various cli…
TaskFusion: Continual Anomaly Detection for Heterogeneous Tabular Data
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））；RPTU Kaiserslautern-Landau（凯泽斯劳滕-兰道大学）；Hochschule Bonn-Rhein-Sieg (H-BRS)（波恩-莱茵-锡格应用技术大学）
Sch of Info & Comm Engin, University of Electronic Science and Technology of China（电子科技大学信息与通信工程学院）；Sch of Comp Sci & Engin, Nanyang Technological University（南洋理工大学计算机科学与工程学院）；Sch of Comp Sci & Tech, Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳）计算机科学与技术学院）
Federated learning (FL) is a rapidly growing privacy-preserving collaborative machine learning paradigm. In practical FL applications, local data from each data silo reflect local usage patterns. Therefore, there exists heterogeneity of data distributions among data owners (a.k.a. FL clients). If not handled properly, this can lead to model performance degradation. This challenge has inspired the research field of heterogeneous federated learning, which currently remains open. In this paper, we propose a data heterogeneity-robust FL approach, FedGSP, to address this challenge by leveraging on …
HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Models
Institute of Space Internet, Fudan University（复旦大学空间互联网研究所）；School of Computer Science, Fudan University（复旦大学计算机科学学院）；Department of Electrical and Electronic Engineering, University of Hong Kong（香港大学电子与电气工程系）；Mohamed bin Zayed University of Artificial Intelligence（莫扎伊德人工智能大学）；Massachusetts Institute of Technology（麻省理工学院）；Data61, CSIRO（CSIRO Data61）；School of Computer Engineering, Nanyang Technological University（南洋理工大学计算机工程学院）
Michigan State University（密歇根州立大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Federated learning (FL) provides a distributed learning framework for multiple participants to collaborate learning without sharing raw data. In many practical FL scenarios, participants have heterogeneous resources due to disparities in hardware and inference dynamics that require quickly loading models of different sizes and levels of robustness. The heterogeneity and dynamics together impose significant challenges to existing FL approaches and thus greatly limit FL's applicability. In this paper, we propose a novel Split-Mix FL strategy for heterogeneous participants that, once training is …
Ping An Technology (Shenzhen) Co., Ltd.（平安科技（深圳）有限公司）
Pre-trained BERT models have achieved impressive performance in many natural language processing (NLP) tasks. However, in many real-world situations, textual data are usually decentralized over many clients and unable to be uploaded to a central server due to privacy protection and regulations. Federated learning (FL) enables multiple clients collaboratively to train a global model while keeping the local data privacy. A few researches have investigated BERT in federated learning setting, but the problem of performance loss caused by heterogeneous (e.g., non-IID) data over clients remain under…
University of Michigan（密歇根大学）；Duke University（杜克大学）
Analysis of heterogeneous patterns in complex spatio-temporal data finds usage across various domains in applied science and engineering, including training autonomous vehicles to navigate in complex traffic scenarios. Motivated by applications arising in the transportation domain, in this paper we develop a model for learning heterogeneous and dynamic patterns of velocity field data. We draw from basic nonparameric Bayesian modeling elements such as hierarchical Dirichlet process and infinite hidden Markov model, while the smoothness of each homogeneous velocity field element is captured with…
Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）；Kaiser Permanente Division of Research（凯撒永久医疗研究部）
We develop an unsupervised probabilistic model for heterogeneous Electronic Health Record (EHR) data. Utilizing a mixture model formulation, our approach directly models sequences of arbitrary length, such as medications and laboratory results. This allows for subgrouping and incorporation of the dynamics underlying heterogeneous data types. The model consists of a layered set of latent variables that encode underlying structure in the data. These variables represent subject subgroups at the top layer, and unobserved states for sequences in the second layer. We train this model on episodic dat…
Iran University of Science and Technology（伊朗科学技术大学）；Institute for Research in Fundamental Sciences (IPM)（基础科学研究所（IPM））；Sharif University of Technology（谢里夫理工大学）
Memory bandwidth bottleneck is a major challenges in processing machine learning (ML) algorithms. In-memory acceleration has potential to address this problem; however, it needs to address two challenges. First, in-memory accelerator should be general enough to support a large set of different ML algorithms. Second, it should be efficient enough to utilize bandwidth while meeting limited power and area budgets of logic layer of a 3D-stacked memory. We observe that previous work fails to simultaneously address both challenges. We propose ORIGAMI, a heterogeneous set of in-memory accelerators, t…
Stanford University（斯坦福大学）；Athinoula A. Martinos Center for Biomedical Imaging（阿西努拉·A·马丁诺斯生物医学成像中心）；Massachusetts General Hospital（马萨诸塞州综合医院）
Federated learning is an emerging research paradigm for enabling collaboratively training deep learning models without sharing patient data. However, the data from different institutions are usually heterogeneous across institutions, which may reduce the performance of models trained using federated learning. In this study, we propose a novel heterogeneity-aware federated learning method, SplitAVG, to overcome the performance drops from data heterogeneity in federated learning. Unlike previous federated methods that require complex heuristic training or hyper parameter tuning, our SplitAVG lev…
HARMONY: Bridging the Personalization-Generalization Gap by Mitigating Representation Skew in Heterogeneous Split Federated Learning
Department of ECE and INMC, Seoul National University, Seoul, South Korea（首尔国立大学电子与信息科学系及网络信息中心，首尔，韩国）；Graduate School of Data Science, Seoul National University, Seoul, South Korea（首尔国立大学数据科学研究生院，首尔，韩国）；Department of Computer Science, University of Colorado Boulder, CO, USA（科罗拉多大学博尔德分校计算机科学系，科罗拉多州，美国）
COMSPLIT: A Communication-Aware Split Learning Design for Heterogeneous IoT Platforms
Faculty of Technical Sciences, University of Novi Sad（诺维萨德大学技术科学学院）；Institute for Artificial Intelligence Research and Development of Serbia（塞尔维亚人工智能研究与发展研究所）；Science, Technology and Innovation Unit, The Abdus Salam International Centre for Theoretical Physics（阿卜杜勒·萨拉姆国际理论物理中心科学、技术与创新部）
Principled Federated Random Forests for Heterogeneous Data
Inria（法国国家信息与自动化研究所）；PreMeDICaL（预医学实验室）；Inserm（法国国家医学研究院）；University of Montpellier（蒙彼利埃大学）；Sorbonne Université（索邦大学）；Université Paris Cité（巴黎城市大学）；CNRS（国家科学研究中心）；LPSM（巴黎高等师范学院）
In this article we propose an optimal method referred to as SPlit for splitting a dataset into training and testing sets. SPlit is based on the method of Support Points (SP), which was initially developed for finding the optimal representative points of a continuous distribution. We adapt SP for subsampling from a dataset using a sequential nearest neighbor algorithm. We also extend SP to deal with categorical variables so that SPlit can be applied to both regression and classification problems. The implementation of SPlit on real datasets shows substantial improvement in the worst-case testin…
RLHGNN: Reinforcement Learning-driven Heterogeneous Graph Neural Network for Next Activity Prediction in Business Processes
College of Computer Science and Technology, Zhejiang University of Technology（浙江工业大学计算机科学与技术学院）；Zhejiang Key Laboratory of Visual Information Intelligent Processing（浙江省视觉信息智能处理重点实验室）；University of Southern Queensland（南方昆士兰大学）
Rakuten Institute of Technology（乐天技术研究所）；Nanyang Technological University（南洋理工大学）
This paper leverages heterogeneous auxiliary information to address the data sparsity problem of recommender systems. We propose a model that learns a shared feature space from heterogeneous data, such as item descriptions, product tags and online purchase history, to obtain better predictions. Our model consists of autoencoders, not only for numerical and categorical data, but also for sequential data, which enables capturing user tastes, item characteristics and the recent dynamics of user preference. We learn the autoencoder architecture for each data source independently in order to better…
University of Cambridge（剑桥大学）；University of Vienna（维也纳大学）；Microsoft Research Cambridge（微软剑桥研究院）
Deep generative models often perform poorly in real-world applications due to the heterogeneity of natural data sets. Heterogeneity arises from data containing different types of features (categorical, ordinal, continuous, etc.) and features of the same type having different marginal distributions. We propose an extension of variational autoencoders (VAEs) called VAEM to handle such heterogeneous data. VAEM is a deep generative model that is trained in a two stage manner such that the first stage provides a more uniform representation of the data to the second stage, thereby sidestepping the p…
