Accelerated Gradient Tracking over Time-varying Graphs for Decentralized Optimization
Institute of Robotics and Automatic Information Systems, College of Artificial Intelligence, Nankai University（南开大学人工智能学院机器人与自动信息系统研究所）；State Key Lab of General AI, School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院通用人工智能省部共建国家重点实验室）
Huan Li, Zhouchen Lin
备注：
Correct one typo in Algorithm 1: $\mathcal{N}_{(i)}$ -> $\mathcal{N}_{(i)}^k$
Decentralized optimization over time-varying graphs has been increasingly common in modern machine learning with massive data stored on millions of mobile devices, such as in federated learning. This paper revisits the widely used accelerated gradient tracking and extends it to time-varying graphs. We prove that the practical single loop accelerated gradient tracking needs $O((\fracγ{1-σ_γ})^2\sqrt{\frac{L}ε})$ and $O((\fracγ{1-σ_γ})^{1.5}\sqrt{\frac{L}μ}\log\frac{1}ε)$ iterations to reach an $ε$-optimal solution over time-varying graphs when the problems are nonstrongly convex and strongly convex, respectively, where $γ$ and $σ_γ$ are two common constants charactering the network connectivity, $L$ and $μ$ are the smoothness and strong convexity constants, respectively, and one iteration corresponds to one gradient oracle call and one communication round. Our convergence rates improve significantly over the ones of $O(\frac{1}{ε^{5/7}})$ and $O((\frac{L}μ)^{5/7}\frac{1}{(1-σ)^{1.5}}\log\frac{1}ε)$, respectively, which were proved in the original literature of accelerated gradient tracking only for static graphs, where $\fracγ{1-σ_γ}$ equals $\frac{1}{1-σ}$ when the network is time-invariant. When combining with a multiple consensus subroutine, the dependence on the network connectivity constants can be further improved to $O(1)$ and $O(\fracγ{1-σ_γ})$ for the gradient oracle and communication round complexities, respectively. When the network is static, by employing the Chebyshev acceleration, our complexities exactly match the lower bounds without hiding any poly-logarithmic factor for both nonstrongly convex and strongly convex problems.
Xi’an Jiaotong University（西安交通大学）；Fudan University（复旦大学）；The Hong Kong Polytechnic University（香港理工大学）；The Hong Kong University of Science and Technology（香港科技大学）
A Variance-Reduced Stochastic Gradient Tracking Algorithm for Decentralized Optimization with Orthogonality Constraints
State Key Laboratory of Scientific and Engineering Computing, Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院科学与工程计算国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）
Compressed Gradient Tracking for Decentralized Optimization Over General Directed Networks
Shanghai Center for Mathematical Sciences, Fudan University（复旦大学上海数学中心）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）；Department of Mathematics, Michigan State University（密歇根州立大学数学系）
Decentralized Optimization on Compact Submanifolds by Quantized Riemannian Gradient Tracking
National Special Education Resource Center for Children with Autism, Zhejiang Normal University（浙江师范大学生自闭症教育资源中心）；School of Computer Science and Technology, Zhejiang Normal University（浙江师范大学计算机科学与技术学院）；China Mobile Research Institute（中国移动研究院）；Institute of Cyber-Systems and Control, Zhejiang University（浙江大学控制系统研究所）；SGIT AI Lab, State Grid Corporation of China（国网SGIT人工智能实验室）；Zhejiang Key Laboratory of Intelligent Education Technology and Application, Zhejiang Normal University（浙江智能教育技术与应用重点实验室）；School of Information Engineering, Huzhou University（湖州大学信息工程学院）；Centre for Frontier Artificial Intelligence Research, A*STAR（A*STAR前沿人工智能研究中心）
DeEPCA: Decentralized Exact PCA with Linear Convergence Rate
Shenzhen Research Institute of Big Data（深圳大数据研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Hong Kong University of Science and Technology（香港科技大学）
SPARKLE: A Unified Single-Loop Primal-Dual Framework for Decentralized Bilevel Optimization
Peking University（北京大学）；IBM Research（IBM研究院）；University of Pennsylvania（宾夕法尼亚大学）；National Engineering Labratory for Big Data Analytics and Applications（国家大数据分析与应用工程实验室）；AI for Science Institute（科学智能研究所）
Distributed Optimization with Streaming Data: A Temporal Weighting Perspective
School of Electrical, Computer and Energy Engineering, Arizona State University（亚利桑那州立大学电气、计算机与能源工程学院）；Department of Electrical Engineering (ISY), Linköping University（林雪平大学电气工程系）
Decentralized Sparse Linear Regression via Gradient-Tracking: Linear Convergence and Statistical Guarantees
Texas A&M University（德克萨斯农工大学）；Purdue University（普渡大学）；The Pennsylvania State University（宾夕法尼亚州立大学）；The University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
DAMA: A Unified Accelerated Approach for Decentralized Nonconvex Minimax Optimization-Part II: Convergence and Performance Analyses
Institute of Electrical and Micro Engineering, École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院电气与微电子研究所）；École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）；Kuwait University（科威特大学）
Sporadic Gradient Tracking over Directed Graphs: A Theoretical Perspective on Decentralized Federated Learning
School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA（电气与计算机工程学院，普渡大学，西拉法基，印第安纳州，美国）；Department of Computer Science and Engineering, Yonsei University, Seoul, South Korea（计算机科学与工程系，延世大学，首尔，韩国）；Department of Electrical Engineering, University at Buffalo -- SUNY, Buffalo, NY, USA（电气工程系，布法罗大学——SUNY，布法罗，纽约州，美国）
Decentralized Federated Learning with Gradient Tracking over Time-Varying Directed Networks
School of Electrical, Computer and Energy Engineering, Arizona State University（亚利桑那州立大学电气、计算机与能源工程学院）；Department of Electrical and Computer Engineering, Princeton University（普林斯顿大学电气与计算机工程系）
Decentralized Nonconvex Composite Federated Learning with Gradient Tracking and Momentum
School of Cyber Science & Engineering, Southeast University（东南大学网络空间科学与工程学院）；Institute of Artificial Intelligence (TeleAI) of China Telecom（中国电信人工智能研究院）；School of Mathematics, Southeast University（东南大学数学学院）；Purple Mountain Laboratories（紫金山实验室）
