arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2312.06887cs.LGcs.AI
收藏

理解并利用神经网络的学习阶段

Understanding and Leveraging the Learning Phases of Neural Networks

University of Liechtenstein(列支敦士登大学);Georgia Institute of Technology(佐治亚理工学院)

Johannes Schneider, Mohit Prabhushankar

AI总结:

该研究针对深度神经网络学习动力学认知不足的问题,通过重构损失分析发现学习存在三个阶段,提出基于经典复杂度分析的研究方法,并得出分类器预训练应早于最优性能停止的迁移学习新实践。

摘要:

深度神经网络的学习动力学尚未得到充分理解。信息瓶颈(information bottleneck, IB)理论曾提出存在独立的拟合阶段和压缩阶段,但此后该观点引发了大量争议。我们通过研究训练过程中参数演化下各层对输入的重构能力与预测性能,全面分析了学习动力学。我们使用ResNet、VGG等常见数据集和架构,通过实验证明存在三个阶段:(i) 重构损失近乎恒定,(ii) 重构损失下降,(iii) 重构损失上升。我们还推导了一个基于实验依据的数据模型,并证明了单层网络中阶段的存在性。从技术层面看,我们的方法利用了经典复杂度分析,它与IB理论的区别在于依赖重构损失测量而非信息论度量,来关联中间层与输入的信息。我们的工作提出了一种迁移学习的新最佳实践:实验表明,分类器的预训练应在其性能达到最优之前很早就停止。

英文摘要:

The learning dynamics of deep neural networks are not well understood. The information bottleneck (IB) theory proclaimed separate fitting and compression phases. But they have since been heavily debated. We comprehensively analyze the learning dynamics by investigating a layer's reconstruction ability of the input and prediction performance based on the evolution of parameters during training. We empirically show the existence of three phases using common datasets and architectures such as ResNet and VGG: (i) near constant reconstruction loss, (ii) decrease, and (iii) increase. We also derive an empirically grounded data model and prove the existence of phases for single-layer networks. Technically, our approach leverages classical complexity analysis. It differs from IB by relying on measuring reconstruction loss rather than information theoretic measures to relate information of intermediate layers and inputs. Our work implies a new best practice for transfer learning: We show empirically that the pre-training of a classifier should stop well before its performance is optimal.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑