The Min-Max Complexity of Distributed Stochastic Convex Optimization with Intermittent Communication

The Minimax Complexity of Distributed Optimization

In this thesis, I study the minimax oracle complexity of distributed stochastic optimization. First, I present the "graph oracle model", an extension of the classic oracle complexity framework that can be applied to study distributed optimization algorithms. Next, I describe a general approach to proving optimization lower bounds for arbitrary randomized algorithms (as opposed to more restricted classes of algorithms, e.g., deterministic or "zero-respecting" algorithms), which is used extensively throughout the thesis. For the remainder of the thesis, I focus on the specific case of the "inter…

用于间歇性通信的分布式异构学习的局部随机梯度下降的局限性与潜力

The Limits and Potentials of Local SGD for Distributed Heterogeneous Learning with Intermittent Communication

研究分布式异构学习中局部随机梯度下降的局限性与潜力，在一阶数据异构性假设下给出其新下界，证明加速小批量SGD的极小极大最优性，考虑高阶假设给出新上界，强调需更好数据异构性模型理解其有效性。

用于具有间歇性策略通信的博弈的解耦 SGDA

Decoupled SGDA for Games with Intermittent Strategy Communication

针对多人博弈中频繁交换策略不可行的问题，提出 Decoupled SGDA 方法，允许玩家基于过时策略独立更新并定期同步，在强凸-强凹及弱耦合博弈中显著降低通信成本并保持近最优收敛率。

间歇性、异步专家反馈对贝叶斯优化有多大用处？

How Useful is Intermittent, Asynchronous Expert Feedback for Bayesian Optimization?

本文研究少量随机到达且非阻塞的专家反馈能否改善贝叶斯优化，提出在BO循环上运行独立线程收集反馈并学习贝叶斯偏好模型，实验表明间歇性异步反馈可提升或约束BO，有助于改进自动驾驶实验室。

Minibatch vs Local SGD for Heterogeneous Distributed Learning

We analyze Local SGD (aka parallel or federated SGD) and Minibatch SGD in the heterogeneous distributed setting, where each machine has access to stochastic gradient estimates for a different, machine-specific, convex objective; the goal is to optimize w.r.t. the average objective; and machines can only communicate intermittently. We argue that, (i) Minibatch SGD (even without acceleration) dominates all existing analysis of Local SGD in this setting, (ii) accelerated Minibatch SGD is optimal when the heterogeneity is high, and (iii) present the first upper bound for Local SGD that improves ov…

Federated Learning Under Intermittent Client Availability and Time-Varying Communication Constraints

Federated learning systems facilitate training of global models in settings where potentially heterogeneous data is distributed across a large number of clients. Such systems operate in settings with intermittent client availability and/or time-varying communication constraints. As a result, the global models trained by federated learning systems may be biased towards clients with higher availability. We propose F3AST, an unbiased algorithm that dynamically learns an availability-dependent client selection strategy which asymptotically minimizes the impact of client-sampling variance on the gl…