AI 中文总结
本文将动态积分二次约束（IQCs）用于加速随机梯度算法的分析与综合，提出刻画小批量梯度二阶统计量的IQCs、验证指数收敛速率的半定规划及界定渐近方差的条件，还提出识别最优小批量梯度算法的凸综合过程。
AI 中文摘要
本文将动态积分二次约束（IQCs）应用于加速随机梯度算法的分析与综合。我们考虑复合目标函数，其梯度可通过小批量采样近似，并将所得随机梯度神谕建模为鲁棒控制中Lur'e系统风格的反馈非线性项。第一项主要贡献是一系列IQCs，用于刻画小批量梯度的二阶统计量，拓展了经典Zames-Falb乘子。第二项贡献是一个半定规划，用于验证随机梯度算法的指数收敛速率，以及一个互补条件，用于界定由非消失梯度噪声引发的渐近方差。第三项贡献是一个凸综合过程，可识别具有最小可验证收敛速率的小批量梯度算法。
英文摘要
This article applies dynamic integral quadratic constraints (IQCs) to the analysis and synthesis of accelerated stochastic gradient algorithms. We consider composite objective functions whose gradient can be approximated via mini-batch sampling and we model the resulting stochastic gradient oracle as a feedback nonlinearity in the spirit of Lur'e systems from robust control. Our first main contribution is a family of IQCs that characterize the second-order statistics of mini-batch gradients, extending the classical Zames--Falb multipliers. Our second contribution is a semidefinite-program for certifying exponential convergence rates of stochastic gradient algorithms, and a complementary condition for bounding the asymptotic variance caused by non-vanishing gradient noise. Our third contribution is a convex synthesis procedure that identifies mini-batch gradient algorithms with the smallest certifiable convergence rate.
