The conjunction of edge intelligence and the ever-growing Internet-of-Things (IoT) network heralds a new era of collaborative machine learning, with federated learning (FL) emerging as the most prominent paradigm. With the growing interest in these learning schemes, researchers started addressing some of their most fundamental limitations. Indeed, conventional FL with a central aggregator presents a single point of failure and a network bottleneck. To bypass this issue, decentralized FL where nodes collaborate in a peer-to-peer network has been proposed. Despite the latter's efficiency, communication costs and data heterogeneity remain key challenges in decentralized FL. In this context, we propose a novel scheme, called opportunistic communication-efficient decentralized federated learning, a.k.a., OCD-FL, consisting of a systematic FL peer selection for collaboration, aiming to achieve maximum FL knowledge gain while reducing energy consumption. Experimental results demonstrate the capability of OCD-FL to achieve similar or better performances than the fully collaborative FL, while significantly reducing consumed energy by at least 30% and up to 80%.
Federated Learning systems use a centralized server to aggregate model updates. This is a bandwidth and resource-heavy constraint and exposes the system to privacy concerns. We instead implement a peer to peer learning system in which nodes train on their own data and periodically perform a weighted average of their parameters with that of their peers according to a learned trust matrix. So far, we have created a model client framework and have been using this to run experiments on the proposed system using multiple virtual nodes which in reality exist on the same computer. We used this strate…
Federated learning (FL) is a privacy-friendly type of machine learning where devices locally train a model on their private data and typically communicate model updates with a server. In decentralized FL (DFL), peers communicate model updates with each other instead. However, DFL is challenging since (1) the training data possessed by different peers is often non-i.i.d. (i.e., distributed differently between the peers) and (2) malicious, or Byzantine, attackers can share arbitrary model updates with other peers to subvert the training process. We address these two challenges and present Bristl…
Evidential Trust-Aware Model Personalization in Decentralized Federated Learning for Wearable IoT
Quantum Cloud Computing and Distributed Systems (qCLOUDS) Lab（量子云计算与分布式系统实验室）；School of Computing and Information Systems（计算与信息系统学院）；The University of Melbourne, Australia（墨尔本大学）
RISE Research Institutes of Sweden（瑞典RISE研究院）；Chalmers University of Technology（查尔姆斯理工大学）；KTH Royal Institute of Technology（KTH皇家理工学院）
We tackle the non-convex problem of learning a personalized deep learning model in a decentralized setting. More specifically, we study decentralized federated learning, a peer-to-peer setting where data is distributed among many clients and where there is no central server to orchestrate the training. In real world scenarios, the data distributions are often heterogeneous between clients. Therefore, in this work we study the problem of how to efficiently learn a model in a peer-to-peer system with non-iid client data. We propose a method named Performance-Based Neighbor Selection (PENS) where…
University of Science and Technology of China（中国科学技术大学）；JD Explore Academy（京东探索研究院）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）；Institute of Artificial Intelligence（人工智能研究院）
Personalized federated learning is proposed to handle the data heterogeneity problem amongst clients by learning dedicated tailored local models for each user. However, existing works are often built in a centralized way, leading to high communication pressure and high vulnerability when a failure or an attack on the central server occurs. In this work, we propose a novel personalized federated learning framework in a decentralized (peer-to-peer) communication protocol named Dis-PFL, which employs personalized sparse masks to customize sparse local models on the edge. To further save the commu…
Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Peng Cheng Laboratory（鹏城实验室）
Swarm learning (SL) is an emerging promising decentralized machine learning paradigm and has achieved high performance in clinical applications. SL solves the problem of a central structure in federated learning by combining edge computing and blockchain-based peer-to-peer network. While there are promising results in the assumption of the independent and identically distributed (IID) data across participants, SL suffers from performance degradation as the degree of the non-IID data increases. To address this problem, we propose a generative augmentation framework in swarm learning called SL-G…
Decentralized Learning (DL) is a peer--to--peer learning approach that allows a group of users to jointly train a machine learning model. To ensure correctness, DL should be robust, i.e., Byzantine users must not be able to tamper with the result of the collaboration. In this paper, we introduce two \textit{new} attacks against DL where a Byzantine user can: make the network converge to an arbitrary model of their choice, and exclude an arbitrary user from the learning process. We demonstrate our attacks' efficiency against Self--Centered Clipping, the state--of--the--art robust DL protocol. F…
Department of Computer, Control and Management Engineering, Sapienza University of Rome（罗马萨皮恩扎大学计算机、控制与管理工程系）；Department of Computer Science, Sapienza University of Rome（罗马萨皮恩扎大学计算机科学系）；Università della Svizzera Italiana（意大利瑞士大学）
Establishing how a set of learners can provide privacy-preserving federated learning in a fully decentralized (peer-to-peer, no coordinator) manner is an open problem. We propose the first privacy-preserving consensus-based algorithm for the distributed learners to achieve decentralized global model aggregation in an environment of high mobility, where the communication graph between the learners may vary between successive rounds of model aggregation. In particular, in each round of global model aggregation, the Metropolis-Hastings method is applied to update the weighted adjacency matrix bas…
PID-Guided Partial Alignment for Multimodal Decentralized Federated Learning
Department of Electrical and Computer Engineering, Stony Brook University（石溪大学电气与计算机工程系）；Department of Applied Mathematics and Statistics and the Department of Computer Science, Stony Brook University（石溪大学应用数学与统计系和计算机科学系）；Department of Electrical and Computer Engineering, New York University（纽约大学电气与计算机工程系）；Department of Computer Science, New York Institute of Technology（纽约理工学院计算机科学系）
Centre for Research & Technology Hellas（希腊研究与技术中心）；Information Technologies Institute（信息技术研究所）
In this work, we aim to classify nodes of unstructured peer-to-peer networks with communication uncertainty, such as users of decentralized social networks. Graph Neural Networks (GNNs) are known to improve the accuracy of simple classifiers in centralized settings by leveraging naturally occurring network links, but graph convolutional layers are challenging to implement in decentralized settings when node neighbors are not constantly available. We address this problem by employing decoupled GNNs, where base classifier predictions and errors are diffused through graphs after training. For the…
Recently, local peer topology has been shown to influence the overall convergence of decentralized learning (DL) graphs in the presence of data heterogeneity. In this paper, we demonstrate the advantages of constructing a proxy-based locally heterogeneous DL topology to enhance convergence and maintain data privacy. In particular, we propose a novel peer clumping strategy to efficiently cluster peers before arranging them in a final training graph. By showing how locally heterogeneous graphs outperform locally homogeneous graphs of similar size and from the same global data distribution, we pr…
Purdue University（普渡大学）；IMEC（IMEC（微电子研究中心））；University of California Berkeley（加州大学伯克利分校）
Current deep learning (DL) systems rely on a centralized computing paradigm which limits the amount of available training data, increases system latency, and adds privacy and security constraints. On-device learning, enabled by decentralized and distributed training of DL models over peer-to-peer wirelessly connected edge devices, not only alleviate the above limitations but also enable next-gen applications that need DL models to continuously interact and learn from their environment. However, this necessitates the development of novel training algorithms that train DL models over time-varyin…
Delft University of Technology（代尔夫特理工大学）；ENSTA Paris（巴黎国立高等先进技术学校）；Institut Polytechnique de Paris（巴黎理工学院）；Shell Global Solutions International B.V.（壳牌全球解决方案国际有限公司）
The Ohio State University（俄亥俄州立大学）；Iowa State University（爱荷华州立大学）；University of Minnesota（明尼苏达大学）；IBM Thomas J. Watson Research Center（IBM托马斯·J·沃森研究中心）
In recent years, decentralized bilevel optimization problems have received increasing attention in the networking and machine learning communities thanks to their versatility in modeling decentralized learning problems over peer-to-peer networks (e.g., multi-agent meta-learning, multi-agent reinforcement learning, personalized training, and Byzantine-resilient learning). However, for decentralized bilevel optimization over peer-to-peer networks with limited computation and communication capabilities, how to achieve low sample and communication complexities are two fundamental challenges that r…
Altruistic Ride Sharing: A Framework for Fair and Sustainable Urban Mobility via Peer-to-Peer Incentives
Dept. of CSIS and APPCAIR, BITS Pilani K K Birla Goa Campus（BITS Pilani K K Birla Goa校区计算机科学与信息系统系及应用人工智能中心）；School of Computer Science, Carnegie Mellon University（卡内基梅隆大学计算机科学学院）
Fully Decentralized Joint Learning of Personalized Models and Collaboration Graphs
GE – Global Research（通用电气全球研究中心）；Inria（法国国家信息与自动化研究所）；Univ Lyon（里昂大学）；UJM-Saint-Etienne（圣艾蒂安让·莫内大学）；CNRS（法国国家科学研究中心）；Laboratoire Hubert Curien（于贝尔·居里安实验室）
RISE Research Institutes of Sweden（瑞典RISE研究院）；KTH Royal Institute of Technology（KTH皇家理工学院）
We study the problem of training personalized deep learning models in a decentralized peer-to-peer setting, focusing on the setting where data distributions differ between the clients and where different clients have different local learning tasks. We study both covariate and label shift, and our contribution is an algorithm which for each client finds beneficial collaborations based on a similarity estimate for the local task. Our method does not rely on hyperparameters which are hard to estimate, such as the number of client clusters, but rather continuously adapts to the network topology us…
Harnessing Implicit Cooperation: A Multi-Agent Reinforcement Learning Approach Towards Decentralized Local Energy Markets
Department of Mechanical Engineering, Universidad de los Andes（机械工程系，andes大学）；Department of Industrial Engineering, Universidad de los Andes（工业工程系，andes大学）
Peking University（北京大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Michigan State University（密歇根州立大学）；Fudan University（复旦大学）；Alibaba DAMO Academy（阿里巴巴达摩院）；Princeton University（普林斯顿大学）
Decentralized optimization is an emerging paradigm in distributed learning in which agents achieve network-wide solutions by peer-to-peer communication without the central server. Since communication tends to be slower than computation, when each agent communicates with only a few neighboring agents per iteration, they can complete iterations faster than with more agents or a central server. However, the total number of iterations to reach a network-wide solution is affected by the speed at which the agents' information is ``mixed'' by communication. We found that popular communication topolog…
Expectation Maximization (EM) is the standard method to learn Gaussian mixtures. Yet its classic, centralized form is often infeasible, due to privacy concerns and computational and communication bottlenecks. Prior work dealt with data distributed by examples, horizontal partitioning, but we lack a counterpart for data scattered by features, an increasingly common scheme (e.g. user profiling with data from multiple entities). To fill this gap, we provide an EM-based algorithm to fit Gaussian mixtures to Vertically Partitioned data (VP-EM). In federated learning setups, our algorithm matches th…
The Ohio State University（俄亥俄州立大学）；Wayne State University（韦恩州立大学）；Air Force Research Laboratory（美国空军研究实验室）
Decentralized bilevel optimization has received increasing attention recently due to its foundational role in many emerging multi-agent learning paradigms (e.g., multi-agent meta-learning and multi-agent reinforcement learning) over peer-to-peer edge networks. However, to work with the limited computation and communication capabilities of edge networks, a major challenge in developing decentralized bilevel optimization techniques is to lower sample and communication complexities. This motivates us to develop a new decentralized bilevel optimization called DIAMOND (decentralized single-timescal…
This paper studies a decentralized stochastic gradient tracking (DSGT) algorithm for non-convex empirical risk minimization problems over a peer-to-peer network of nodes, which is in sharp contrast to the existing DSGT only for convex problems. To ensure exact convergence and handle the variance among decentralized datasets, each node performs a stochastic gradient (SG) tracking step by using a mini-batch of samples, where the batch size is designed to be proportional to the size of the local dataset. We explicitly evaluate the convergence rate of DSGT with respect to the number of iterations …
Federated learning (FL) has been facilitating privacy-preserving deep learning in many walks of life such as medical image classification, network intrusion detection, and so forth. Whereas it necessitates a central parameter server for model aggregation, which brings about delayed model communication and vulnerability to adversarial attacks. A fully decentralized architecture like Swarm Learning allows peer-to-peer communication among distributed nodes, without the central server. One of the most challenging issues in decentralized deep learning is that data owned by each node are usually non…
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Google Inc.（谷歌公司）；University of California, Santa Barbara（加利福尼亚大学圣巴巴拉分校）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）
Decentralized SGD is an emerging training method for deep learning known for its much less (thus faster) communication per iteration, which relaxes the averaging step in parallel SGD to inexact averaging. The less exact the averaging is, however, the more the total iterations the training needs to take. Therefore, the key to making decentralized SGD efficient is to realize nearly-exact averaging using little communication. This requires a skillful choice of communication topology, which is an under-studied topic in decentralized optimization. In this paper, we study so-called exponential graph…
