arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.20724cond-mat.dis-nncs.AI
收藏

通过自旋玻璃理论视角探索损失景观

Exploring Loss Landscapes through the Lens of Spin Glass Theory

College of Computer Science and Software Engineering, Shenzhen University(深圳大学计算机与软件学院);The Education University of Hong Kong(香港教育大学)

Hao Liao, Wei Zhang, Zhanyi Huang, Zexiao Long, Mingyang Zhou, Xiaoqun Wu, Rui Mao, Chi Ho Yeung

AI总结:

本研究从统计物理自旋玻璃理论视角出发,通过四种方案探究ReLU单隐藏层DNN的损失景观结构与特性,验证其与自旋玻璃的相似性,并证实损失景观平坦极小值可提升模型泛化性。

摘要:

过去十年,深度学习取得了重大进展,催生了众多突破性应用。尽管有这些进步,人们对深度学习的高泛化性——尤其是在过参数化空间中的高泛化性——的理解仍然有限。例如,在深度神经网络(DNN)中,其内部表征、决策机制、过参数化空间中不出现过拟合的特性、优异的泛化性等问题仍未得到充分理解。成功的应用往往被视为经验成果而非科学成就。本文以统计物理学中的自旋玻璃(一种具有复杂能量景观和大量亚稳态的系统)为新视角,深入研究DNN的损失景观,以理解DNN的工作原理。我们研究了采用修正线性单元(ReLU)激活的单隐藏层神经网络的损失景观,并提出了若干方案来检验DNN与自旋玻璃之间的相似性。具体而言,我们使用:(1)在DNN参数空间中进行随机游走,以揭示其损失景观的结构;(2)置换-插值方案,研究由隐藏层置换对称性导致的损失景观中相同区域副本之间的联系;(3)层次聚类,揭示DNN训练得到的解之间的层级结构,这让人联想到自旋玻璃中所谓的复本对称破缺(RSB)现象(即Parisi解);(4)最后,我们检验了DNN损失景观的崎岖度与其泛化性之间的关系,证明了平坦极小值能提升泛化性能。

英文摘要:

In the past decade, significant strides in deep learning have led to numerous groundbreaking applications. Despite these advancements, the understanding of the high generalizability of deep learning, especially in such an over-parametrized space, remains limited. For instance, in deep neural networks (DNNs), their internal representations, decision-making mechanism, absence of overfitting in an over-parametrized space, superior generalizability, etc., remain less understood. Successful applications are often considered as empirical rather than scientific achievement. This paper delves into the loss landscape of DNNs through the lens of spin glass in statistical physics, a system characterized by a complex energy landscape with numerous metastable states, as a novel perspective in understanding how DNNs work. We investigated the loss landscape of single hidden layer neural networks activated by Rectified Linear Unit (ReLU) function, and introduced several protocols to examine the analogy between DNNs and spin glass. Specifically, we used (1) random walk in the parameter space of DNNs to unravel the structures in their loss landscape; (2) a permutation-interpolation protocol to study the connection between copies of identical regions in the loss landscape due to the permutation symmetry in the hidden layers; (3) hierarchical clustering to reveal the hierarchy among trained solutions of DNNs, reminiscent of the so-called Replica Symmetry Breaking (RSB) phenomenon (i.e. the Parisi solution) in spin glass; (4) finally, we examine the relationship between the ruggedness of DNN's loss landscape and its generalizability, showing an improvement of flattened minima.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑