arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.09184stat.MLcs.LG
收藏

深度学习模型的双尺度复杂度度量

A Two-Scale Complexity Measure for Deep Learning Models

Department of Mathematics, University of Trento(特伦托大学数学系);DSH, Bruno Kessler Fondation(DSH布鲁诺·凯斯勒基金会);Department of Mathematics, ETH(苏黎世联邦理工学院数学系);IBM Quantum, IBM Research Europe(IBM量子计算,IBM欧洲研究院)

Massimiliano Datres, Gian Paolo Leonardi, Alessio Figalli, David Sutter

AI总结:

本文提出基于有效维度的容量度量 2sED,可证明界定泛化误差,并通过逐层迭代近似使其适用于大规模深度学习模型,实验显示其与训练误差相关且近似效果良好。

摘要:

我们基于有效维度,为统计模型引入一种新的容量度量 2sED。在对模型的温和假设下,这一新量可证明地界定泛化误差。此外,在标准数据集和流行模型架构上的仿真表明,2sED 与训练误差具有良好的相关性。对于马尔可夫模型,我们展示了如何通过逐层迭代方法从下方高效近似 2sED,这使我们能够处理具有大量参数的深度学习模型。仿真结果表明,对于不同的主流模型和数据集,该近似效果良好。

英文摘要:

We introduce a novel capacity measure 2sED for statistical models based on the effective dimension. The new quantity provably bounds the generalization error under mild assumptions on the model. Furthermore, simulations on standard data sets and popular model architectures show that 2sED correlates well with the training error. For Markovian models, we show how to efficiently approximate 2sED from below through a layerwise iterative approach, which allows us to tackle deep learning models with a large number of parameters. Simulation results suggest that the approximation is good for different prominent models and data sets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑