Distributed optimization with open collaboration is a popular field since it provides an opportunity for small groups/companies/universities, and individuals to jointly solve huge-scale problems. However, standard optimization algorithms are fragile in such settings due to the possible presence of so-called Byzantine workers -- participants that can send (intentionally or not) incorrect information instead of the one prescribed by the protocol (e.g., send anti-gradient instead of stochastic gradients). Thus, the problem of designing distributed methods with provable robustness to Byzantine workers has been receiving a lot of attention recently. In particular, several works consider a very promising way to achieve Byzantine tolerance via exploiting variance reduction and robust aggregation. The existing approaches use SAGA- and SARAH-type variance-reduced estimators, while another popular estimator -- SVRG -- is not studied in the context of Byzantine-robustness. In this work, we close this gap in the literature and propose a new method -- Byzantine-Robust Loopless Stochastic Variance Reduced Gradient (BR-LSVRG). We derive non-asymptotic convergence guarantees for the new method in the strongly convex case and compare its performance with existing approaches in numerical experiments.
Prox-DBRO-VR: A Unified Analysis on Byzantine-Resilient Decentralized Stochastic Composite Optimization with Variance Reduction and Non-Asymptotic Convergence Rates
Central South University（中南大学）；City University of Hong Kong（香港城市大学）；University of New South Wales（新南威尔士大学）；Southwest University（西南大学）
Unbiased and Biased Variance-Reduced Forward-Reflected-Backward Splitting Methods for Stochastic Composite Inclusions
Department of Statistics and Operations Research（统计与运筹学系）；The University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Hanes Hall, UNC-Chapel Hill, NC 27599-3260（UNC-教堂山分校318号哈尼斯楼，NC 27599-3260）
Resilient Decentralized Preconditioned Stochastic Proximal Optimization with Gradient Sketching over Time-Varying Networks
Hubei Normal University（湖北师范大学）；City University of Hong Kong（香港城市大学）；University of New South Wales（新南威尔士大学）；Southwest University（西南大学）；Shenzhen University of Advanced Technology（深圳先进技术研究院）
DAMA: A Unified Accelerated Approach for Decentralized Nonconvex Minimax Optimization-Part II: Convergence and Performance Analyses
Institute of Electrical and Micro Engineering, École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院电气与微电子研究所）；École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）；Kuwait University（科威特大学）
Towards closing the gap between the theory and practice of SVRG
Télécom Paris（巴黎电信学院）；IPP（巴黎理工学院）；Princeton University（普林斯顿大学）；INRIA（法国国家信息与自动化研究所）；Ecole Normale Supérieure（巴黎高等师范学院）；PSL Research University（巴黎文理研究大学）
最早用于解决经验风险最小化问题的方差缩减随机方法之一是SVRG方法（Johnson & Zhang 2013）。SVRG是基于内外循环的方法，其中在外循环中评估参考全梯度，之后执行内循环的$m \in \mathbb{N}$步骤，其中参考梯度用于构建当前梯度的方差减小估计。SVRG方法及其分析的简单性导致了甚至非凸优化的多种扩展和变体。我们提供了一个更一般的分析SVRG比以前做的任意采样，这使我们能够分析几乎所有形式的迷你通过一个单一的定理。此外，我们的分析重点关注SVRG的更实用变体，包括无环SVRG的新变体（Hofman et al 2015，Kovalev et al 2019，Kulunchakov and Mairal 2019）和k-SVRG的变体（Raj and Stich 2018），其中$m=n$和$n$是数据点的数量。由于我们的设置和分析反映了在实践中所做的，我们能够设置的参数，如小批量大小和步长使用我们的理论在这样一种方式，产生一个更有效的算法在实践中，我们在广泛的数值实验中显示。
Stochastic Optimisation Framework using the Core Imaging Library and Synergistic Image Reconstruction Framework for PET Reconstruction
Finden Ltd（芬登有限公司）；Rutherford Appleton Laboratory（卢瑟福·阿普尔顿实验室）；Science & Technology Facilities Council（科学与技术设施委员会）；National Physical Laboratory（国家物理实验室）；Électricité de France（法国电力公司）；CSIRO（联邦科学与工业研究组织）；Technical University of Denmark（丹麦技术大学）；University College London（伦敦大学学院）；KU Leuven（鲁汶大学）；Institute of Nuclear Medicine, University College London（伦敦大学学院核医学研究所）
INTERACT: Achieving Low Sample and Communication Complexities in Decentralized Bilevel Learning over Networks
The Ohio State University（俄亥俄州立大学）；Iowa State University（爱荷华州立大学）；University of Minnesota（明尼苏达大学）；IBM Thomas J. Watson Research Center（IBM托马斯·J·沃森研究中心）
