Distributed optimization methods for large-scale machine learning suffer from a communication bottleneck. It is difficult to reduce this bottleneck while still efficiently and accurately aggregating partial work from different machines. In this paper, we present a novel generalization of the recent communication-efficient primal-dual framework (CoCoA) for distributed optimization. Our framework, CoCoA+, allows for additive combination of local updates to the global parameters at each iteration, whereas previous schemes with convergence guarantees only allow conservative averaging. We give stronger (primal-dual) convergence rate guarantees for both CoCoA as well as our new variants, and generalize the theory for both methods to cover non-smooth convex loss functions. We provide an extensive experimental comparison that shows the markedly improved performance of CoCoA+ on several real-world distributed datasets, especially when scaling up the number of machines.
Gradient Descent Averaging and Primal-dual Averaging for Strongly Convex Optimization
Institute of Evaluation and Assessment Research, Academy of Military Science（军事科学院评估论证研究中心）；Command and Control Engineering College, Army Engineering University（陆军工程大学指挥控制工程学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
On the Effectiveness of Mode Exploration in Bayesian Model Averaging for Neural Networks
Massachusetts Institute of Technology, Lincoln Laboratory（麻省理工学院林肯实验室）
已经出现了在监督学习设置中使用深度神经网络来产生校准的预测概率的多种技术，这些技术利用方法来集成在循环训练或从多个随机起始点（深度集成）进行训练期间发现的不同解决方案。然而，只有有限数量的工作已经调查了探索每个不同的解决方案（后验模式）周围的局部区域的效用。我们在CIFAR-10数据集上使用三种众所周知的深度架构，评估了几种简单的方法，用于探索权重空间的局部区域，包括Brier评分、准确性和预期校准误差。我们考虑贝叶斯推理技术（变分推理和应用于softmax输出层的Hamilton Monte Carlo）以及利用接近最优的随机梯度下降轨迹。虽然将单独的模式添加到合奏统一提高性能，我们表明，这里考虑的简单模式探索方法产生很少或没有改善没有模式探索合奏。
