亚线性结构化深度神经网络实现组合函数的特征学习一致性
Sublinearly Structured Deep Neural Networks Achieve Feature Learning Consistency for Compositional Functions
AI总结：
针对亚线性结构化DNN（输入/输出维度和隐层神经元数随样本量亚线性增长），证明了学习层次组合函数时的特征学习一致性，并在过参数化下仍成立；实验表明其预测性能不逊于宽网络，且常用CNN（如AlexNet）具有亚线性结构。
摘要：
在过去十年中，深度神经网络（DNN）在复杂机器学习任务上取得了显著成功，但其性能的理论基础仍不完整。从统计角度看，一个自然的问题是：DNN能否达到与经典模型相当的特征学习和预测一致性？虽然完整刻画尚未解决，但我们为一大类子类提供了正面结果。我们建立了亚线性结构化DNN（即输入/输出维度和隐层神经元数随样本量亚线性增长的架构）在学习层次组合目标函数时的特征学习一致性保证。重要的是，即使在传统“过参数化”区域（参数总数超过训练样本数）中，这种一致性仍然成立。实验上，亚线性结构化DNN在预测中匹配或超越宽DNN。结构审计进一步表明，广泛使用的卷积神经网络（CNN），包括AlexNet、VGGNet、ResNet、GoogLeNet，在其图像分类基准上具有亚线性结构。我们还证明了亚线性结构化DNN在大样本极限下能够实现层次组合函数的通用逼近。此外，图像本身具有固有的层次组合结构。综上所述，这些结果从统计角度解释了为什么许多大规模深度学习模型在大量图像数据集上经过充分训练后能够成功。
英文摘要：
Over the past decade, deep neural networks (DNNs) have achieved remarkable success on complex machine-learning tasks, yet the theoretical foundations of their performance remain incomplete. From a statistical viewpoint, a natural question is: can DNNs attain feature-learning and prediction consistency comparable to that of classical models? While a full characterization is open, we provide positive results for a broad subclass. We establish feature-learning consistency guarantees for sublinearly structured DNNs-architectures whose input/output dimensions and number of hidden neurons grow sublinearly with the sample size-when learning hierarchically compositional target functions. Importantly, this consistency still holds even in the conventional "over-parameterized" regime where the total number of parameters exceeds the number of training samples. Empirically, sublinearly structured DNNs match or surpass wide DNNs in prediction. A structural audit further indicates that widely used convolutional neural networks (CNNs), including AlexNet, VGGNet, ResNet, GoogLeNet, are sublinearly structured on their image classification benchmarks. We further prove that the sublinearly structured DNNs achieve universal approximation for hierarchically compositional functions in the large-sample limit. Moreover, images exhibit an inherent hierarchical, compositional structure. Taken together, these results explain, through a statistical lens, why many large-scale deep learning models succeed after adequate training on massive image datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
