模块临界性在深度网络推广中的有趣作用
The intriguing role of module criticality in the generalization of deep networks
摘要:
我们研究了深度神经网络(DNN)的某些模块比其他模块更关键的现象。这意味着将它们的参数值倒回初始化,同时保持其他模块固定在训练参数上,会导致网络性能大幅下降。我们的分析揭示了损失景观的有趣特性,这使得我们提出了一种称为模块临界度的复杂性度量,该度量基于连接模块参数的初始值和最终值的山谷的形状。我们制定如何泛化与模块的关键性,并表明,这种措施是能够解释优越的泛化性能的一些架构比其他人,而早期的措施未能做到这一点。
英文摘要:
We study the phenomenon that some modules of deep neural networks (DNNs) are more critical than others. Meaning that rewinding their parameter values back to initialization, while keeping other modules fixed at the trained parameters, results in a large drop in the network's performance. Our analysis reveals interesting properties of the loss landscape which leads us to propose a complexity measure, called module criticality, based on the shape of the valleys that connects the initial and final values of the module parameters. We formulate how generalization relates to the module criticality, and show that this measure is able to explain the superior generalization performance of some architectures over others, whereas earlier measures fail to do so.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。