Study of Robust Distributed Diffusion RLS Algorithms with Side Information for Adaptive Networks
Southwest University of Science and Technology（西南科技大学）；Southwest Jiaotong University（西南交通大学）；CETUC, PUC-Rio（里约热内卢天主教大学CETUC研究中心）；University of York（约克大学）；Sichuan University（四川大学）
Y. Yu, H. Zhao, R. C. de Lamare, Y. Zakharov, L. Lu
This work develops robust diffusion recursive least squares algorithms to mitigate the performance degradation often experienced in networks of agents in the presence of impulsive noise. The first algorithm minimizes an exponentially weighted least-squares cost function subject to a time-dependent constraint on the squared norm of the intermediate update at each node. A recursive strategy for computing the constraint is proposed using side information from the neighboring nodes to further improve the robustness. We also analyze the mean-square convergence behavior of the proposed algorithm. The second proposed algorithm is a modification of the first one based on the dichotomous coordinate descent iterations. It has a performance similar to that of the former, however its complexity is significantly lower especially when input regressors of agents have a shift structure and it is well suited to practical implementation. Simulations show the superiority of the proposed algorithms over previously reported techniques in various impulsive noise scenarios.
Centre for Telecommunications Studies, Pontifical Catholic University of Rio de Janeiro（里约热内卢天主教大学电信研究中心）；University of York（约克大学）；University of São Paulo（圣保罗大学）
In this work, we present an energy-efficient distributed learning framework using coarsely quantized signals for Internet of Things (IoT) networks. In particular, we develop a distributed quantization-aware recursive least squares (DQA-RLS) algorithm that can learn parameters in an energy-efficient fashion using signals quantized with few bits while requiring a low computational cost. Numerical results assess the DQA-RLS algorithm against existing techniques for a distributed parameter estimation task where IoT devices operate in a peer-to-peer mode.
Most kernel-based methods, such as kernel or Gaussian process regression, kernel PCA, ICA, or $k$-means clustering, do not scale to large datasets, because constructing and storing the kernel matrix $\mathbf{K}_n$ requires at least $\mathcal{O}(n^2)$ time and space for $n$ samples. Recent works show that sampling points with replacement according to their ridge leverage scores (RLS) generates small dictionaries of relevant points with strong spectral approximation guarantees for $\mathbf{K}_n$. The drawback of RLS-based methods is that computing exact RLS requires constructing and storing the …
Multiple Kronecker RLS fusion-based link propagation for drug-side effect prediction
University of Electronic Science and Technology of China（电子科技大学）；University of Nottingham Ningbo（宁波诺丁汉大学）；Halmstad University（哈尔姆斯塔德大学）；Yangtze Delta Region Institute (Quzhou), University of Electronic Science and Technology of China（电子科技大学长三角研究院（衢州））
National University of Singapore（新加坡国立大学）；Indian Institute of Science（印度科学学院）；Google Research（谷歌研究院）
Communication efficient distributed mean estimation is an important primitive that arises in many distributed learning and optimization scenarios such as federated learning. Without any probabilistic assumptions on the underlying data, we study the problem of distributed mean estimation where the server has access to side information. We propose \emph{Wyner-Ziv estimators}, which are communication and computationally efficient and near-optimal when an upper bound for the distance between the side information and the data is known. As a corollary, we also show that our algorithms provide effici…
William & Mary（威廉与玛丽学院）；Old Dominion University（欧道明大学）
As the number of IoT devices has increased rapidly, IoT botnets have exploited the vulnerabilities of IoT devices. However, it is still challenging to detect the initial intrusion on IoT devices prior to massive attacks. Recent studies have utilized power side-channel information to identify this intrusion behavior on IoT devices but still lack accurate models in real-time for ubiquitous botnet detection. We proposed the first online intrusion detection system called DeepAuditor for IoT devices via power auditing. To develop the real-time system, we proposed a lightweight power auditing device…
Distributed Computing Laboratory (DCL), École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院（EPFL）分布式计算实验室（DCL））
Momentum is a variant of gradient descent that has been proposed for its benefits on convergence. In a distributed setting, momentum can be implemented either at the server or the worker side. When the aggregation rule used by the server is linear, commutativity with addition makes both deployments equivalent. Robustness and privacy are however among motivations to abandon linear aggregation rules. In this work, we demonstrate the benefits on robustness of using momentum at the worker side. We first prove that computing momentum at the workers reduces the variance-norm ratio of the gradient es…
German University in Cairo（开罗德国大学）；Ain Shams University（艾因夏姆斯大学）
Brain tumors are one of the life-threatening forms of cancer. Previous studies have classified brain tumors using deep neural networks. In this paper, we perform the later task using a collaborative deep learning technique, more specifically split learning. Split learning allows collaborative learning via neural networks splitting into two (or more) parts, a client-side network and a server-side network. The client-side is trained to a certain layer called the cut layer. Then, the rest of the training is resumed on the server-side network. Vertical distribution, a method for distributing data …
School of Engineering and Applied Sciences, Harvard University（哈佛大学工程与应用科学学院）；School of Electrical, Computer and Energy Engineering, Arizona State University（亚利桑那州立大学电气、计算机与能源工程学院）
We consider large scale distributed optimization over a set of edge devices connected to a central server, where the limited communication bandwidth between the server and edge devices imposes a significant bottleneck for the optimization procedure. Inspired by recent advances in federated learning, we propose a distributed stochastic gradient descent (SGD) type algorithm that exploits the sparsity of the gradient, when possible, to reduce communication burden. At the heart of the algorithm is to use compressed sensing techniques for the compression of the local stochastic gradients at the dev…
Federated Learning (FL), Split Learning (SL), and SplitFed Learning (SFL) are three recent developments in distributed machine learning that are gaining attention due to their ability to preserve the privacy of raw data. Thus, they are widely applicable in various domains where data is sensitive, such as large-scale medical image classification, internet-of-medical-things, and cross-organization phishing email detection. SFL is developed on the confluence point of FL and SL. It brings the best of FL and SL by providing parallel client-side machine learning model updates from the FL paradigm an…
BOSS ZhiPin Career Science Lab (CSL)（BOSS直聘职业科学实验室（CSL））；La Trobe University（拉筹伯大学）
Distributed deep learning systems (DDLS) train deep neural network models by utilizing the distributed resources of a cluster. Developers of DDLS are required to make many decisions to process their particular workloads in their chosen environment efficiently. The advent of GPU-based deep learning, the ever-increasing size of datasets and deep neural network models, in combination with the bandwidth constraints that exist in cluster environments require developers of DDLS to be innovative in order to train high quality models quickly. Comparing DDLS side-by-side is difficult due to their exten…
University of Rochester（罗切斯特大学）；ETH Zurich（苏黎世联邦理工学院）
Scalable and efficient distributed learning is one of the main driving forces behind the recent rapid advancement of machine learning and artificial intelligence. One prominent feature of this topic is that recent progresses have been made by researchers in two communities: (1) the system community such as database, data management, and distributed systems, and (2) the machine learning and mathematical optimization community. The interaction and knowledge sharing between these two communities has led to the rapid development of new distributed learning systems and theory. In this work, we hope…
Imperial College London（帝国理工学院）；Sharif University of Technology（谢里夫理工大学）；New York University（纽约大学）；EPFL（洛桑联邦理工学院）
We present a novel deep neural network (DNN) architecture for compressing an image when a correlated image is available as side information only at the decoder. This problem is known as distributed source coding (DSC) in information theory. In particular, we consider a pair of stereo images, which generally have high correlation with each other due to overlapping fields of view, and assume that one image of the pair is to be compressed and transmitted, while the other image is available only at the decoder. In the proposed architecture, the encoder maps the input image to a latent space, quant…
University of Tennessee, Knoxville（田纳西大学诺克斯维尔分校）；Intuit AI Research（财捷人工智能研究院）
Distributed machine learning paradigms, such as federated learning, have been recently adopted in many privacy-critical applications for speech analysis. However, such frameworks are vulnerable to privacy leakage attacks from shared gradients. Despite extensive efforts in the image domain, the exploration of speech privacy leakage from gradients is quite limited. In this paper, we explore methods for recovering private speech/speaker information from the shared gradients in distributed learning settings. We conduct experiments on a keyword spotting model with two different types of speech feat…
FedZMG: Efficient Client-Side Optimization in Federated Learning
TelSiP Research Laboratory, Department of Electrical and Electronic Engineering, School of Engineering, University of West Attica（TelSiP研究实验室，电气与电子工程系，工程学院，西阿提卡大学）
The standard model of online prediction deals with serial processing of inputs by a single processor. However, in large-scale online prediction problems, where inputs arrive at a high rate, an increasingly common necessity is to distribute the computation across several processors. A non-trivial challenge is to design distributed algorithms for online prediction, which maintain good regret guarantees. In \cite{DMB}, we presented the DMB algorithm, which is a generic framework to convert any serial gradient-based online prediction algorithm into a distributed algorithm. Moreover, its regret gua…
