Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices

基于分位数的损失过滤用于抗离群值随机梯度下降

Quantile-based Loss Filtering for Outlier-Robust Stochastic Gradient Descent

本研究提出分位数-k损失SGD（QkL-SGD）框架，通过采样损失并选择较低分位数索引更新，在凸假设下证明线性收敛，实验显示中间分位数优于标准SGD和min-k-loss，兼具鲁棒性与信息量。

Stochastic Gradient Variance Reduction by Solving a Filtering Problem

Deep neural networks (DNN) are typically optimized using stochastic gradient descent (SGD). However, the estimation of the gradient using stochastic samples tends to be noisy and unreliable, resulting in large gradient variance and bad convergence. In this paper, we propose \textbf{Filter Gradient Decent}~(FGD), an efficient stochastic optimization algorithm that makes the consistent estimation of the local gradient by solving an adaptive filtering problem with different design of filters. Our method reduces variance in stochastic gradient descent by incorporating the historical states to enha…

元目标引导的偏标签学习消歧

Meta Objective Guided Disambiguation for Partial Label Learning

本文提出元目标引导消歧（MoGD）框架，通过小验证集上的元损失重加权候选标签，以加权交叉熵训练分类器，解决偏标签学习中缺乏先验和早期预测不可靠的问题，理论证明收敛性并在基准和真实数据集上取得先进性能。

Communication-Efficient Robust Federated Learning Over Heterogeneous Datasets

This work investigates fault-resilient federated learning when the data samples are non-uniformly distributed across workers, and the number of faulty workers is unknown to the central server. In the presence of adversarially faulty workers who may strategically corrupt datasets, the local messages exchanged (e.g., local gradients and/or local model parameters) can be unreliable, and thus the vanilla stochastic gradient descent (SGD) algorithm is not guaranteed to converge. Recently developed algorithms improve upon vanilla SGD by providing robustness to faulty workers at the price of slowing …

Federated Learning over Wireless Networks: A Band-limited Coordinated Descent Approach

We consider a many-to-one wireless architecture for federated learning at the network edge, where multiple edge devices collaboratively train a model using local data. The unreliable nature of wireless connectivity, together with constraints in computing resources at edge devices, dictates that the local updates at edge devices should be carefully crafted and compressed to match the wireless communication resources available and should work in concert with the receiver. Thus motivated, we propose SGD-based bandlimited coordinate descent algorithms for such settings. Specifically, for the wirel…