Random Reshuffling (RR), also known as Stochastic Gradient Descent (SGD) without replacement, is a popular and theoretically grounded method for finite-sum minimization. We propose two new algorithms: Proximal and Federated Random Reshuffing (ProxRR and FedRR). The first algorithm, ProxRR, solves composite convex finite-sum minimization problems in which the objective is the sum of a (potentially non-smooth) convex regularizer and an average of $n$ smooth objectives. We obtain the second algorithm, FedRR, as a special case of ProxRR applied to a reformulation of distributed problems with either homogeneous or heterogeneous data. We study the algorithms' convergence properties with constant and decreasing stepsizes, and show that they have considerable advantages over Proximal and Local SGD. In particular, our methods have superior complexities and ProxRR evaluates the proximal operator once per epoch only. When the proximal operator is expensive to compute, this small difference makes ProxRR up to $n$ times faster than algorithms that evaluate the proximal operator in every iteration. We give examples of practical optimization tasks where the proximal operator is difficult to compute and ProxRR has a clear advantage. Finally, we corroborate our results with experiments on real data sets.
Beijing Institute of Technology（北京理工大学）；Beijing Institute of Technology Chongqing Innovation Center（北京理工大学重庆创新中心）；Tongji University（同济大学）；Nanyang Technological University（南洋理工大学）
The non-smooth finite-sum minimization is a fundamental problem in machine learning. This paper develops a distributed stochastic proximal-gradient algorithm with random reshuffling to solve the finite-sum minimization over time-varying multi-agent networks. The objective function is a sum of differentiable convex functions and non-smooth regularization. Each agent in the network updates local variables with a constant step-size by local information and cooperates to seek an optimal solution. We prove that local variable estimates generated by the proposed algorithm achieve consensus and are a…
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
Random Reshuffling (RR), which is a variant of Stochastic Gradient Descent (SGD) employing sampling without replacement, is an immensely popular method for training supervised machine learning models via empirical risk minimization. Due to its superior practical performance, it is embedded and often set as default in standard machine learning software. Under the name FedRR, this method was recently shown to be applicable to federated learning (Mishchenko et al.,2021), with superior performance when compared to common baselines such as Local SGD. Inspired by this development, we design three ne…
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人研究院）；Shenzhen Research Institute of Big Data（深圳大数据研究院）
We study the random reshuffling (RR) method for smooth nonconvex optimization problems with a finite-sum structure. Though this method is widely utilized in practice such as the training of neural networks, its convergence behavior is only understood in several limited settings. In this paper, under the well-known Kurdyka-Lojasiewicz (KL) inequality, we establish strong limit-point convergence results for RR with appropriate diminishing step sizes, namely, the whole sequence of iterates generated by RR is convergent and converges to a single stationary point in an almost sure sense. In additio…
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Moscow Institute of Physics and Technology（莫斯科物理技术学院）；Mila, Université de Montréal（蒙特利尔大学米拉研究所）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Princeton University（普林斯顿大学）
Gradient compression is a popular technique for improving communication complexity of stochastic first-order methods in distributed training of machine learning models. However, the existing works consider only with-replacement sampling of stochastic gradients. In contrast, it is well-known in practice and recently confirmed in theory that stochastic methods based on without-replacement sampling, e.g., Random Reshuffling (RR) method, perform better than ones that sample the gradients with-replacement. In this work, we close this gap in the literature and provide the first analysis of methods w…
We present a theoretical study of server-side optimization in federated learning. Our results are the first to show that the widely popular heuristic of scaling the client updates with an extra parameter is very useful in the context of Federated Averaging (FedAvg) with local passes over the client data. Each local pass is performed without replacement using Random Reshuffling, which is a key reason we can show improved complexities. In particular, we prove that whenever the local stepsizes are small, and the update direction is given by FedAvg in conjunction with Random Reshuffling over all c…
In distributed learning, local SGD (also known as federated averaging) and its simple baseline minibatch SGD are widely studied optimization methods. Most existing analyses of these methods assume independent and unbiased gradient estimates obtained via with-replacement sampling. In contrast, we study shuffling-based variants: minibatch and local Random Reshuffling, which draw stochastic gradients without replacement and are thus closer to practice. For smooth functions satisfying the Polyak-Łojasiewicz condition, we obtain convergence bounds (in the large epoch regime) which show that these s…
CMAP, École polytechnique, Insitut Polytechnique de Paris（CMAP，巴黎高等理工学院，巴黎理工 institute）；LAAS-CNRS, Université de Toulouse, CNRS, Toulouse, France（LAAS-CNRS，图卢兹大学，CNRS，法国图卢兹）；Department of SEEM, The Chinese University of Hong Kong Shatin, Hong Kong（信息工程系，香港中文大学（深圳））
The practice of applying several local updates before aggregation across clients has been empirically shown to be a successful approach to overcoming the communication bottleneck in Federated Learning (FL). Such methods are usually implemented by having clients perform one or more epochs of local training per round while randomly reshuffling their finite dataset in each epoch. Data imbalance, where clients have different numbers of local training samples, is ubiquitous in FL applications, resulting in different clients performing different numbers of local updates in each round. In this work, …
In this paper, we study communication efficient distributed algorithms for distributionally robust federated learning via periodic averaging with adaptive sampling. In contrast to standard empirical risk minimization, due to the minimax structure of the underlying optimization problem, a key difficulty arises from the fact that the global parameter that controls the mixture of local losses can only be updated infrequently on the global stage. To compensate for this, we propose a Distributionally Robust Federated Averaging (DRFA) algorithm that employs a novel snapshotting scheme to approximate…
Indian Institute of Technology Kanpur（印度理工学院坎普尔分校）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Google Research（谷歌研究院）
We analyze the convergence rates of stochastic gradient algorithms for smooth finite-sum minimax optimization and show that, for many such algorithms, sampling the data points without replacement leads to faster convergence compared to sampling with replacement. For the smooth and strongly convex-strongly concave setting, we consider gradient descent ascent and the proximal point method, and present a unified analysis of two popular without-replacement sampling strategies, namely Random Reshuffling (RR), which shuffles the data every epoch, and Single Shuffling or Shuffle Once (SO), which shuf…
Compressed Proximal Federated Learning for Non-Convex Composite Optimization on Heterogeneous Data
School of Intelligent Systems Engineering, Sun Yat-Sen University（中山大学智能系统工程学院）；Department of Automation, Beijing National Research Center for Information Science and Technology, Tsinghua University（清华大学自动化系）；Department of Mechanical Engineering, University of Victoria（维多利亚大学机械工程系）
Traditionally, clustered federated learning groups clients with the same data distribution into a cluster, so that every client is uniquely associated with one data distribution and helps train a model for this distribution. We relax this hard association assumption to soft clustered federated learning, which allows every local dataset to follow a mixture of multiple source distributions. We propose FedSoft, which trains both locally personalized models and high-quality cluster models in this setting. FedSoft limits client workload by using proximal updates to require the completion of only on…
FedCanon: Non-Convex Composite Federated Learning with Efficient Proximal Operation on Heterogeneous Data
School of Cyber Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；National Center for Applied Mathematics in Jiangsu, Southeast University（江苏应用数学中心）；School of Engineering, RMIT University（皇家墨尔本理工大学工程学院）；School of Automation, Southeast University（东南大学自动化学院）
FedSOL: Stabilized Orthogonal Learning with Proximal Restrictions in Federated Learning
Graduate School of AI KAIST（韩国科学技术院人工智能研究生院）；Department of Electrical and Computer Engineering, UBC（不列颠哥伦比亚大学电气与计算机工程系）；Samsung Advanced Institute of Technology（三星先进技术研究院）
Federated Binary Matrix Factorization using Proximal Optimization
KTH Royal Institute of Technology（瑞典皇家理工学院）；CISPA Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心）；Institute for AI in Medicine, UK Essen（埃森大学医院人工智能医学研究所）；Ruhr University Bochum（波鸿鲁尔大学）；Monash University（莫纳什大学）
CNRS（法国国家科学研究中心）；ENS（巴黎高等师范学院）；Inria Sierra（法国国家信息与自动化研究所Sierra项目组）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；CISPA Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心）
We introduce ProxSkip -- a surprisingly simple and provably efficient method for minimizing the sum of a smooth ($f$) and an expensive nonsmooth proximable ($ψ$) function. The canonical approach to solving such problems is via the proximal gradient descent (ProxGD) algorithm, which is based on the evaluation of the gradient of $f$ and the prox operator of $ψ$ in each iteration. In this work we are specifically interested in the regime in which the evaluation of prox is costly relative to the evaluation of the gradient, which is the case in many applications. ProxSkip allows for the expensive p…
Proximal Control of UAVs with Federated Learning for Human-Robot Collaborative Domains
Lab of Systems Engineering and Robotics (LASER), Department of Computer Systems, Universidade Federal da Paraiba（系统工程与机器人实验室（LASER），计算机系统系，巴伊帕巴联邦大学）；Department of Cybernetics, Czech Technical University in Prague（控制论系，布拉格捷克技术大学）
Distributed optimization: designed for federated learning
School of Management（管理学院）；Jinan University（暨南大学）；Guangdong International Cooperation Base of Science and Technology for GBA Smart Logistics（粤港澳大湾区智能物流科技国际合作基地）；School of Intelligent Systems Science and Engineering（智能系统科学与工程学院）；Institute of Physical Internet（物理互联网研究院）；School of Mechanical and Electrical Engineering（机械与电子工程学院）；Jiangxi University of Science and Technology（江西理工大学）；Department of Industrial and Systems Engineering（工业与系统工程系）；The Hong Kong Polytechnic University（香港理工大学）
Non-convex composite federated learning with heterogeneous data
School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology（KTH皇家理工学院电气与计算机科学学院）；Massachusetts General Hospital and Harvard Medical School, Harvard University（哈佛大学麻省总医院及哈佛医学院）
We propose an asymptotic framework to analyze the performance of (personalized) federated learning algorithms. In this new framework, we formulate federated learning as a multi-criterion objective, where the goal is to minimize each client's loss using information from all of the clients. We analyze a linear regression model where, for a given client, we may theoretically compare the performance of various algorithms in the high-dimensional asymptotic limit. This asymptotic multi-criterion approach naturally models the high-dimensional, many-device nature of federated learning. These tools mak…
We study a family of algorithms, which we refer to as local update methods, generalizing many federated and meta-learning algorithms. We prove that for quadratic models, local update methods are equivalent to first-order optimization on a surrogate loss we exactly characterize. Moreover, fundamental algorithmic choices (such as learning rates) explicitly govern a trade-off between the condition number of the surrogate loss and its alignment with the true loss. We derive novel convergence rates showcasing these trade-offs and highlight their importance in communication-limited settings. Using t…
University of Central Florida（中佛罗里达大学）；University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）；Tulane University（杜兰大学）
Federated learning (FL) is a promising strategy for performing privacy-preserving, distributed learning with a network of clients (i.e., edge devices). However, the data distribution among clients is often non-IID in nature, making efficient optimization difficult. To alleviate this issue, many FL algorithms focus on mitigating the effects of data heterogeneity across clients by introducing a variety of proximal terms, some incurring considerable compute and/or memory overheads, to restrain local updates with respect to the global model. Instead, we consider rethinking solutions to data hetero…
In this paper, we propose decentralized and scalable algorithms for Gaussian process (GP) training and prediction in multi-agent systems. To decentralize the implementation of GP training optimization algorithms, we employ the alternating direction method of multipliers (ADMM). A closed-form solution of the decentralized proximal ADMM is provided for the case of GP hyper-parameter training with maximum likelihood estimation. Multiple aggregation techniques for GP prediction are decentralized with the use of iterative and consensus methods. In addition, we propose a covariance-based nearest nei…
Virginia Tech（弗吉尼亚理工大学）；Princeton University（普林斯顿大学）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Future Network of Intelligence Institute（未来网络智能研究院）；Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；University of Oulu（奥卢大学）
A new federated learning (FL) framework enabled by large-scale wireless connectivity is proposed for designing the autonomous controller of connected and autonomous vehicles (CAVs). In this framework, the learning models used by the controllers are collaboratively trained among a group of CAVs. To capture the varying CAV participation in the FL training process and the diverse local data quality among CAVs, a novel dynamic federated proximal (DFP) algorithm is proposed that accounts for the mobility of CAVs, the wireless fading channels, as well as the unbalanced and nonindependent and identic…
Traditionally, federated learning (FL) aims to train a single global model while collaboratively using multiple clients and a server. Two natural challenges that FL algorithms face are heterogeneity in data across clients and collaboration of clients with {\em diverse resources}. In this work, we introduce a \textit{quantized} and \textit{personalized} FL algorithm QuPeD that facilitates collective (personalized model compression) training via \textit{knowledge distillation} (KD) among clients who have access to heterogeneous data and resources. For personalization, we allow clients to learn \…
Traditionally, federated learning (FL) aims to train a single global model while collaboratively using multiple clients and a server. Two natural challenges that FL algorithms face are heterogeneity in data across clients and collaboration of clients with {\em diverse resources}. In this work, we introduce a \textit{quantized} and \textit{personalized} FL algorithm QuPeL that facilitates collective training with heterogeneous clients while respecting resource diversity. For personalization, we allow clients to learn \textit{compressed personalized models} with different quantization parameters…
The University of British Columbia（不列颠哥伦比亚大学）；University of Minnesota（明尼苏达大学）；Rensselaer Polytechnic Institute（伦斯勒理工学院）；Shenzhen University（深圳大学）
This work investigates fault-resilient federated learning when the data samples are non-uniformly distributed across workers, and the number of faulty workers is unknown to the central server. In the presence of adversarially faulty workers who may strategically corrupt datasets, the local messages exchanged (e.g., local gradients and/or local model parameters) can be unreliable, and thus the vanilla stochastic gradient descent (SGD) algorithm is not guaranteed to converge. Recently developed algorithms improve upon vanilla SGD by providing robustness to faulty workers at the price of slowing …
Data used to train machine learning models can be adversarial--maliciously constructed by adversaries to fool the model. Challenge also arises by privacy, confidentiality, or due to legal constraints when data are geographically gathered and stored across multiple learners, some of which may hold even an "anonymized" or unreliable dataset. In this context, the distributionally robust optimization framework is considered for training a parametric model, both in centralized and federated learning settings. The objective is to endow the trained model with robustness against adversarially manipula…
Robust and Personalized Federated Learning for Aircraft-Engine Prognostics under Benign and Adversarial Client Heterogeneity
Rajshahi University of Engineering & Technology（拉杰沙希工程与技术大学）；North South University（北南大学）；Missouri State University（密苏里州立大学）；Malardalen University（梅拉达伦大学）；American International University-Bangladesh（孟加拉国美国国际大学）
