神经网络和非凸压缩感知中Optimization的松弛论证
A Relaxation Argument for Optimization in Neural Networks and Non-Convex Compressed Sensing
摘要：
在实际应用和理论分析中已经观察到，过度参数化有助于在神经网络训练中找到好的极小值。类似地，在本文中，我们研究通过松弛参数来扩展和深化神经网络，以便扩展后的网络足够丰富，可以并行运行原始网络部分的$r$副本，而不必像在过度参数化的场景中那样实现零训练误差。部分副本可以针对层宽度$θ$以$r^θ$可能的方式组合。因此，扩大的网络可以潜在地实现$r^θ$随机初始化的最佳训练误差，但是这是否可以通过梯度下降或类似的训练方法来实现还不清楚。通过引入类似的分层结构，可以将相同的构造应用于其他最优化问题。我们将这个想法应用到非凸压缩感知，在那里我们表明，在某些情况下，我们可以实现的$r^θ$倍增加的机会，以获得一个全局最优值的尺寸$rθ$的凸优化问题。
英文摘要：
It has been observed in practical applications and in theoretical analysis that over-parametrization helps to find good minima in neural network training. Similarly, in this article we study widening and deepening neural networks by a relaxation argument so that the enlarged networks are rich enough to run $r$ copies of parts of the original network in parallel, without necessarily achieving zero training error as in over-parametrized scenarios. The partial copies can be combined in $r^θ$ possible ways for layer width $θ$. Therefore, the enlarged networks can potentially achieve the best training error of $r^θ$ random initializations, but it is not immediately clear if this can be realized via gradient descent or similar training methods. The same construction can be applied to other optimization problems by introducing a similar layered structure. We apply this idea to non-convex compressed sensing, where we show that in some scenarios we can realize the $r^θ$ times increased chance to obtain a global optimum by solving a convex optimization problem of dimension $rθ$.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
