人工神经网络的一个容量缩放定律
A Capacity Scaling Law for Artificial Neural Networks
摘要：
我们推导出预测感知器网络行为的两个临界数的计算。首先，我们推导出我们称之为无损存储器（LM）维度的计算。LM维是Vapnik-Chervonenkis（VC）维的推广，它避免了结构化数据，因此提供了一个完美拟合几乎任何训练数据的上限。其次，我们推导出所谓的麦凯（MK）维度。此限制表示无法训练给定函数的概率为50%。我们的推导是通过将神经网络嵌入到香农的通信模型中来执行的，该模型允许将这两点解释为以比特为单位测量的容量。我们提出了一个证明和实际的实验，使用不同的网络配置，不同的实现，不同的激活函数，和几个学习算法的可重复实验，验证我们的上限。底线是两个容量点的比例与权重的数量严格线性。在其他实际应用中，我们的结果允许比较和基准不同的神经网络实现独立的具体学习任务。我们的研究结果提供了深入了解神经网络的能力和局限性，并为实验设计决策提供了有价值的知识。
英文摘要：
We derive the calculation of two critical numbers predicting the behavior of perceptron networks. First, we derive the calculation of what we call the lossless memory (LM) dimension. The LM dimension is a generalization of the Vapnik--Chervonenkis (VC) dimension that avoids structured data and therefore provides an upper bound for perfectly fitting almost any training data. Second, we derive what we call the MacKay (MK) dimension. This limit indicates a 50% chance of not being able to train a given function. Our derivations are performed by embedding a neural network into Shannon's communication model which allows to interpret the two points as capacities measured in bits. We present a proof and practical experiments that validate our upper bounds with repeatable experiments using different network configurations, diverse implementations, varying activation functions, and several learning algorithms. The bottom line is that the two capacity points scale strictly linear with the number of weights. Among other practical applications, our result allows to compare and benchmark different neural network implementations independent of a concrete learning task. Our results provide insight into the capabilities and limits of neural networks and generate valuable know how for experimental design decisions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
