随机优化梯度压缩的极限
Limits on Gradient Compression for Stochastic Optimization
摘要：
我们考虑随机优化$\ell_p$空间使用访问一阶预言。我们问：{Oracle输出保持不受限制的收敛速度所需的最小精度是多少？}我们的特点是每个$p\geq 1$推导信息理论的下限，并提供量化器，（几乎）实现这些下限的精度。我们的量化器是新的，易于实施。特别是，我们的结果是准确的$p=2$和$p=\infty$，显示在这些设置中所需的最低精度分别是$Θ(d)$和$Θ(\log d)$。后一个结果是令人惊讶的，因为恢复梯度向量将需要$Ω(d)$位。
英文摘要：
We consider stochastic optimization over $\ell_p$ spaces using access to a first-order oracle. We ask: {What is the minimum precision required for oracle outputs to retain the unrestricted convergence rates?} We characterize this precision for every $p\geq 1$ by deriving information theoretic lower bounds and by providing quantizers that (almost) achieve these lower bounds. Our quantizers are new and easy to implement. In particular, our results are exact for $p=2$ and $p=\infty$, showing the minimum precision needed in these settings are $Θ(d)$ and $Θ(\log d)$, respectively. The latter result is surprising since recovering the gradient vector will require $Ω(d)$ bits.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
