基于代理草图和尺度正则化的去偏分布式二阶优化
Debiasing Distributed Second Order Optimization with Surrogate Sketching and Scaled Regularization
摘要：
在分布式二阶优化中，标准策略是平均许多局部估计，每个估计都基于一小部分或一批数据。然而，相对于所有数据的完整解，每台机器上的局部估计通常是有偏差的，这可能会限制平均的有效性。在这里，我们介绍了一种新的技术去偏的局部估计，这导致了理论和经验的改进，分布二阶方法的收敛速度。我们的技术有两个新的组成部分：（1）修改标准草图技术，以获得我们所谓的代理草图;（2）仔细缩放局部计算的全局正则化参数。我们的代理草图是基于行列式点过程，一个家庭的分布，逆海森估计的偏差可以精确计算。基于此计算，我们表明，当目标被最小化是$l_2$ -正则化参数$λ$和个人的机器都给出了一个草图的大小$m$，然后消除偏差，局部估计应使用收缩正则化参数计算由$λ^{\prime}=λ\cdot(1-\frac{d_λ}{m})$，其中$d_λ$是$λ$ -有效尺寸的海森（或二次问题，数据矩阵）。
英文摘要：
In distributed second order optimization, a standard strategy is to average many local estimates, each of which is based on a small sketch or batch of the data. However, the local estimates on each machine are typically biased, relative to the full solution on all of the data, and this can limit the effectiveness of averaging. Here, we introduce a new technique for debiasing the local estimates, which leads to both theoretical and empirical improvements in the convergence rate of distributed second order methods. Our technique has two novel components: (1) modifying standard sketching techniques to obtain what we call a surrogate sketch; and (2) carefully scaling the global regularization parameter for local computations. Our surrogate sketches are based on determinantal point processes, a family of distributions for which the bias of an estimate of the inverse Hessian can be computed exactly. Based on this computation, we show that when the objective being minimized is $l_2$-regularized with parameter $λ$ and individual machines are each given a sketch of size $m$, then to eliminate the bias, local estimates should be computed using a shrunk regularization parameter given by $λ^{\prime}=λ\cdot(1-\frac{d_λ}{m})$, where $d_λ$ is the $λ$-effective dimension of the Hessian (or, for quadratic problems, the data matrix).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
