arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2207.08437math.OCcs.NAmath.NA
收藏

摆脱约束并重新参数化:NNLS与隐式偏差研究

Get rid of your constraints and reparametrize: A study in NNLS and implicit bias

Hung-Hsu Chou, Johannes Maly, Claudio Mayrink Verdun, Bernardo Freitas Paulo da Costa, Heudson Mirandola

AI总结:

本文将过参数化梯度下降与黎曼优化联系起来,通过重新参数化求解非负最小二乘问题,证明梯度流全局收敛并给出离散收敛速率,还提出二阶加速方法及稳定性分析。

摘要:

在过去几年中,人们对理解梯度下降优化的隐式偏差及其与过参数化神经网络泛化性质之间的联系产生了浓厚兴趣。多项工作观察到,当在回归任务的平方损失上训练线性对角网络(对应于过参数化线性回归)时,梯度下降会收敛到特殊解,例如非负解。我们将这一观察与黎曼优化联系起来,并将具有相同初始化的过参数化梯度下降视为黎曼梯度下降。我们利用这一事实来求解非负最小二乘(NNLS),这是许多技术(例如非负矩阵分解)背后的重要问题。我们证明,在重新参数化的目标上的梯度流全局收敛到NNLS解,并为其离散化对应方法提供了收敛速率。与以往方法不同,我们不依赖于指数映射或测地线的计算。我们进一步展示了使用二阶常微分方程的加速收敛,这适用于加速下降方法。最后,我们建立了对负扰动的稳定性,并讨论了向其他约束优化问题的推广。

英文摘要:

Over the past years, there has been significant interest in understanding the implicit bias of gradient descent optimization and its connection to the generalization properties of overparametrized neural networks. Several works observed that when training linear diagonal networks on the square loss for regression tasks (which corresponds to overparametrized linear regression) gradient descent converges to special solutions, e.g., non-negative ones. We connect this observation to Riemannian optimization and view overparametrized GD with identical initialization as a Riemannian GD. We use this fact for solving non-negative least squares (NNLS), an important problem behind many techniques, e.g., non-negative matrix factorization. We show that gradient flow on the reparametrized objective converges globally to NNLS solutions, providing convergence rates also for its discretized counterpart. Unlike previous methods, we do not rely on the calculation of exponential maps or geodesics. We further show accelerated convergence using a second-order ODE, lending itself to accelerated descent methods. Finally, we establish the stability against negative perturbations and discuss generalization to other constrained optimization problems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑