arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2112.09741cs.LGcond-mat.dis-nncond-mat.stat-mechcs.CVstat.ML
收藏

展望未来的深度学习理论:一些基本概念与特征

Envisioning Future Deep Learning Theories: Some Basic Concepts and Characteristics

University of Pennsylvania(宾夕法尼亚大学)

Weijie J. Su

AI总结:

本文提出未来深度学习理论应具备层级架构、随机梯度迭代参数和压缩式信息演化三大特征,并构建图模型 neurashed 以解释隐式正则化、信息瓶颈和局部弹性等经验现象。

摘要:

为推动未来十年深度学习方法的发展,需要一个用于推理现代神经网络的理论框架。尽管揭示深度学习为何如此有效的努力正在增加,但仍缺乏一幅全面的图景,这表明更好的理论是可能的。我们认为,未来的深度学习理论应继承三个特征:层级结构化的网络架构、使用基于随机梯度的方法迭代优化的参数,以及以压缩方式演化的来自数据的信息。作为一种实例化,我们将这些特征整合到一个名为 neurashed 的图模型中。该模型有效解释了深度学习中的一些常见经验模式。特别是,neurashed 使人们能够深入理解隐式正则化、信息瓶颈和局部弹性。最后,我们讨论 neurashed 如何指导深度学习理论的发展。

英文摘要:

To advance deep learning methodologies in the next decade, a theoretical framework for reasoning about modern neural networks is needed. While efforts are increasing toward demystifying why deep learning is so effective, a comprehensive picture remains lacking, suggesting that a better theory is possible. We argue that a future deep learning theory should inherit three characteristics: a \textit{hierarchically} structured network architecture, parameters \textit{iteratively} optimized using stochastic gradient-based methods, and information from the data that evolves \textit{compressively}. As an instantiation, we integrate these characteristics into a graphical model called \textit{neurashed}. This model effectively explains some common empirical patterns in deep learning. In particular, neurashed enables insights into implicit regularization, information bottleneck, and local elasticity. Finally, we discuss how neurashed can guide the development of deep learning theories.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑