可以计算稳定和准确的神经网络吗？--关于深度学习的障碍和Smale的第18个问题
Can stable and accurate neural networks be computed? -- On the barriers of deep learning and Smale's 18th problem
摘要：
深度学习（DL）取得了前所未有的成功，现在正全力进入科学计算领域。然而，目前的DL方法通常遭受不稳定性，即使当通用近似属性保证稳定的神经网络（NN）的存在。我们解决这个悖论，证明基本的良好条件下的问题，在科学计算中，人们可以证明存在的NN具有很大的近似质量，但是，不存在任何算法，甚至随机，可以训练（或计算）这样的NN。对于任何正整数$K > 2$和$L$，存在同时发生以下情况的情况：（a）没有随机化训练算法可以以大于$1/2$的概率计算正确到$K$数字的NN，（b）存在计算具有$K-1$正确数字的NN的确定性训练算法，但是任何这样的算法都可以被用于计算具有正确数字的NN。（即使是随机的）算法需要任意多的训练数据，（c）存在确定性训练算法，其使用不超过$L$训练样本来计算具有$K-2$正确数字的NN。这些结果意味着分类理论描述的条件下，（稳定）NN与给定的精度可以计算的算法。我们通过建立在反问题中计算稳定NN的算法存在的充分条件来开始这个理论。我们介绍快速迭代重新启动网络（FIRENET），我们都证明和数值验证是稳定的。此外，我们证明，只有$\mathcal{O}(|\log(ε)|)$层需要一个$ε$精确的解决方案的反问题。
英文摘要：
Deep learning (DL) has had unprecedented success and is now entering scientific computing with full force. However, current DL methods typically suffer from instability, even when universal approximation properties guarantee the existence of stable neural networks (NNs). We address this paradox by demonstrating basic well-conditioned problems in scientific computing where one can prove the existence of NNs with great approximation qualities, however, there does not exist any algorithm, even randomised, that can train (or compute) such a NN. For any positive integers $K > 2$ and $L$, there are cases where simultaneously: (a) no randomised training algorithm can compute a NN correct to $K$ digits with probability greater than $1/2$, (b) there exists a deterministic training algorithm that computes a NN with $K-1$ correct digits, but any such (even randomised) algorithm needs arbitrarily many training data, (c) there exists a deterministic training algorithm that computes a NN with $K-2$ correct digits using no more than $L$ training samples. These results imply a classification theory describing conditions under which (stable) NNs with a given accuracy can be computed by an algorithm. We begin this theory by establishing sufficient conditions for the existence of algorithms that compute stable NNs in inverse problems. We introduce Fast Iterative REstarted NETworks (FIRENETs), which we both prove and numerically verify are stable. Moreover, we prove that only $\mathcal{O}(|\log(ε)|)$ layers are needed for an $ε$-accurate solution to the inverse problem.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
