arXiv 1706.016862017-06math.OCcs.LGstat.ML

有限和Optimization的方差缩减和加速格式的局限性

Limitations on Variance-Reduction and Acceleration Schemes for Finite Sum Optimization

Weizmann Institute of Science（魏茨曼科学研究所）

Yossi Arjevani

摘要：

我们研究的条件下，能够有效地应用方差减少和加速计划有限和最优化问题。首先，我们表明，也许令人惊讶的是，有限和结构本身，是不足以获得$\tilde{\cO}((n+L/μ)\ln(1/ε))$的复杂性界的$L$ -光滑和$μ$ -强凸的个别功能-还必须知道，个别功能是指在每次迭代的甲骨文。接下来，我们证明了，对于广泛的一类一阶和坐标下降有限和算法（包括，例如，SDCA，SVRG，SAG），不可能得到$\tilde{\cO}((n+\sqrt{n L/μ})\ln(1/ε))$的“加速”复杂性界，除非明确给出强凸性参数。最后，我们表明，当这类算法用于最小化$L$ -光滑和凸有限和，最佳的复杂性界是$\tilde{\cO}(n+L/ε)$，假设（平均）相同的更新规则在每次迭代中使用，和$\tilde{\cO}(n+\sqrt{nL/ε})$，否则。

英文摘要：

We study the conditions under which one is able to efficiently apply variance-reduction and acceleration schemes on finite sum optimization problems. First, we show that, perhaps surprisingly, the finite sum structure by itself, is not sufficient for obtaining a complexity bound of $\tilde{\cO}((n+L/μ)\ln(1/ε))$ for $L$-smooth and $μ$-strongly convex individual functions - one must also know which individual function is being referred to by the oracle at each iteration. Next, we show that for a broad class of first-order and coordinate-descent finite sum algorithms (including, e.g., SDCA, SVRG, SAG), it is not possible to get an `accelerated' complexity bound of $\tilde{\cO}((n+\sqrt{n L/μ})\ln(1/ε))$, unless the strong convexity parameter is given explicitly. Lastly, we show that when this class of algorithms is used for minimizing $L$-smooth and convex finite sums, the optimal complexity bound is $\tilde{\cO}(n+L/ε)$, assuming that (on average) the same update rule is used in every iteration, and $\tilde{\cO}(n+\sqrt{nL/ε})$, otherwise.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

摘要：

英文摘要：

相关论文