深度神经网络的扩展临界区
Extended critical regimes of deep neural networks
摘要:
深度神经网络(DNN)已经成功地应用于许多现实世界的问题,但仍然缺乏对其动力学和计算原理的完整理解。分析DNN的传统理论框架通常假设随机网络具有服从高斯统计的耦合权重。然而,非高斯重尾耦合是DNN中普遍存在的现象。在这里,通过将重尾随机矩阵和非平衡统计物理的理论编织在一起,我们为DNN开发了一种新型的平均场理论,该理论预测重尾权重能够在没有微调参数的情况下出现扩展的临界状态。在这种扩展的临界状态下,DNN表现出丰富而复杂的跨层传播动力学。我们进一步阐明,扩展的临界性赋予DNN深刻的计算优势:平衡内部神经表征的收缩和扩展,并加快训练过程,从而为设计高效的神经架构提供理论指导。
英文摘要:
Deep neural networks (DNNs) have been successfully applied to many real-world problems, but a complete understanding of their dynamical and computational principles is still lacking. Conventional theoretical frameworks for analysing DNNs often assume random networks with coupling weights obeying Gaussian statistics. However, non-Gaussian, heavy-tailed coupling is a ubiquitous phenomenon in DNNs. Here, by weaving together theories of heavy-tailed random matrices and non-equilibrium statistical physics, we develop a new type of mean field theory for DNNs which predicts that heavy-tailed weights enable the emergence of an extended critical regime without fine-tuning parameters. In this extended critical regime, DNNs exhibit rich and complex propagation dynamics across layers. We further elucidate that the extended criticality endows DNNs with profound computational advantages: balancing the contraction as well as expansion of internal neural representations and speeding up training processes, hence providing a theoretical guide for the design of efficient neural architectures.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。