arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1910.13593cs.LGstat.ML
收藏

多任务深度神经分类器中的泛化:统计物理方法

Generalization in multitask deep neural classifiers: a statistical physics approach

Intel AI Lab(英特尔人工智能实验室)

Tyler Lee, Anthony Ndirango

摘要:

正确理解深度神经网络惊人的泛化能力是一个持久的难题。最近,越来越多的基于数值的理论工作为深度神经网络的学习理论提供了重要的见解。最近也有兴趣将这些分析扩展到理解多任务学习如何进一步提高深度神经网络的泛化能力。这些研究几乎只涉及回归任务,这些任务适用于现有的分析技术。我们开发了深度神经网络泛化的非线性动力学分析理论,该神经网络经过训练,使用softmax输出和交叉熵损失解决分类任务,解决单任务和多任务设置。我们通过调整无序系统的统计物理学技术来做到这一点,同时考虑到有限大小的数据集和由训练动态引起的相关输出。我们讨论了我们的理论结果相比,一套全面的数值实验的有效性。我们的分析提供了理论支持的直觉,多任务学习的性能是由任务的噪声和如何以及他们的输入功能相互对齐。高度相关的干净任务彼此受益,而不相关的干净任务可能对个人任务绩效有害。

英文摘要:

A proper understanding of the striking generalization abilities of deep neural networks presents an enduring puzzle. Recently, there has been a growing body of numerically-grounded theoretical work that has contributed important insights to the theory of learning in deep neural nets. There has also been a recent interest in extending these analyses to understanding how multitask learning can further improve the generalization capacity of deep neural nets. These studies deal almost exclusively with regression tasks which are amenable to existing analytical techniques. We develop an analytic theory of the nonlinear dynamics of generalization of deep neural networks trained to solve classification tasks using softmax outputs and cross-entropy loss, addressing both single task and multitask settings. We do so by adapting techniques from the statistical physics of disordered systems, accounting for both finite size datasets and correlated outputs induced by the training dynamics. We discuss the validity of our theoretical results in comparison to a comprehensive suite of numerical experiments. Our analysis provides theoretical support for the intuition that the performance of multitask learning is determined by the noisiness of the tasks and how well their input features align with each other. Highly related, clean tasks benefit each other, whereas unrelated, clean tasks can be detrimental to individual task performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑