arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2203.00246cs.LGcs.AIstat.ML
收藏

监督学习的信息论框架

An Information-Theoretic Framework for Supervised Learning

Stanford University(斯坦福大学)

Hong Jun Jeon, Yifan Zhu, Benjamin Van Roy

摘要:

每年,深度学习都会通过更深、更广的神经网络展示新的、改进的经验结果。同时,在现有的理论框架下,很难分析超过两层的网络,而不依靠计数参数或遇到指数深度的样本复杂度界限。也许尝试从不同的角度分析现代机器学习可能会有成果。在本文中,我们提出了一种新颖的信息理论框架,具有自己的遗憾和样本复杂性概念,用于分析机器学习的数据需求。在我们的框架中,我们首先通过一些经典的例子,如标量估计和线性回归来建立直觉并介绍一般技术。然后,我们使用该框架来研究从具有ReLU激活单元的深度神经网络生成的数据中学习的样本复杂度。对于一个特定的先验分布的权重,我们建立样本的复杂性界限,同时宽度独立和线性的深度。这种先验分布产生了高维的潜在表示,具有很高的概率,承认合理准确的低维近似。最后,我们通过随机单隐层神经网络的实验分析证实了我们的理论结果。

英文摘要:

Each year, deep learning demonstrates new and improved empirical results with deeper and wider neural networks. Meanwhile, with existing theoretical frameworks, it is difficult to analyze networks deeper than two layers without resorting to counting parameters or encountering sample complexity bounds that are exponential in depth. Perhaps it may be fruitful to try to analyze modern machine learning under a different lens. In this paper, we propose a novel information-theoretic framework with its own notions of regret and sample complexity for analyzing the data requirements of machine learning. With our framework, we first work through some classical examples such as scalar estimation and linear regression to build intuition and introduce general techniques. Then, we use the framework to study the sample complexity of learning from data generated by deep neural networks with ReLU activation units. For a particular prior distribution on weights, we establish sample complexity bounds that are simultaneously width independent and linear in depth. This prior distribution gives rise to high-dimensional latent representations that, with high probability, admit reasonably accurate low-dimensional approximations. We conclude by corroborating our theoretical results with experimental analysis of random single-hidden-layer neural networks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑