后向特征修正：深度学习如何实现深度（层级）学习
Backward Feature Correction: How Deep Learning Performs Deep (Hierarchical) Learning
AI总结：
本文提出“后向特征修正”原理，从理论上证明超常数层神经网络可通过 SGD 在多项式时间内高效学习某些高次多项式概念类，而逐层训练、核方法等非层级算法无法做到，揭示了深度学习实现层级学习的核心机制。
摘要：
深度学习也被称为层级学习，其中学习器通过将复杂的目标函数分解为一系列更简单的函数来学习表示该函数，以降低样本复杂度和时间复杂度。本文形式化地分析了多层神经网络如何通过训练目标上的 SGD 高效且自动地执行这种层级学习。在概念层面，我们给出了理论刻画，说明某些类型的深度（即超常数层）神经网络在某些层级任务上仍然可以在样本和时间上被高效训练，而已知没有任何现有算法（包括逐层训练、核方法等）能够高效完成。我们建立了一个名为“后向特征修正”的新原理，即低层特征中的误差在与更高层一起训练时可以被自动修正。我们认为这是深度学习实现深度（层级）学习的关键，而非逐层学习或模拟某种非层级方法。在技术层面，我们证明对于每个输入维度 $d > 0$，存在一个次数为 $ω(1)$ 的多元多项式的概念类，使得使用 $ω(1)$ 层神经网络作为学习器时，SGD 可以在 $\mathsf{poly}(d)$ 时间内学习该类中的任意函数，达到任意 $\frac{1}{\mathsf{poly}(d)}$ 的误差，其方式是通过“后向特征修正”学习将其表示为 $ω(1)$ 层二次函数的复合。相比之下，我们不知道任何其他更简单的算法（包括逐层训练、顺序应用核方法、训练两层网络等）能够在 $\mathsf{poly}(d)$ 时间内学习该概念类，即使误差仅为 $d^{-0.01}$。作为附带结果，我们证明了若干非层级学习器（包括任意核方法）的 $d^{ω(1)}$ 下界。
英文摘要：
Deep learning is also known as hierarchical learning, where the learner _learns_ to represent a complicated target function by decomposing it into a sequence of simpler functions to reduce sample and time complexity. This paper formally analyzes how multi-layer neural networks can perform such hierarchical learning _efficiently_ and _automatically_ by SGD on the training objective. On the conceptual side, we present a theoretical characterizations of how certain types of deep (i.e. super-constant layer) neural networks can still be sample and time efficiently trained on some hierarchical tasks, when no existing algorithm (including layerwise training, kernel method, etc) is known to be efficient. We establish a new principle called "backward feature correction", where the errors in the lower-level features can be automatically corrected when training together with the higher-level layers. We believe this is a key behind how deep learning is performing deep (hierarchical) learning, as opposed to layerwise learning or simulating some non-hierarchical method. On the technical side, we show for every input dimension $d > 0$, there is a concept class of degree $ω(1)$ multi-variate polynomials so that, using $ω(1)$-layer neural networks as learners, SGD can learn any function from this class in $\mathsf{poly}(d)$ time to any $\frac{1}{\mathsf{poly}(d)}$ error, through learning to represent it as a composition of $ω(1)$ layers of quadratic functions using "backward feature correction." In contrast, we do not know any other simpler algorithm (including layerwise training, applying kernel method sequentially, training a two-layer network, etc) that can learn this concept class in $\mathsf{poly}(d)$ time even to any $d^{-0.01}$ error. As a side result, we prove $d^{ω(1)}$ lower bounds for several non-hierarchical learners, including any kernel methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
