基于超球能量最小化的神经网络正则化
Regularizing Neural Networks via Minimizing Hyperspherical Energy
摘要：
受物理学中汤姆逊问题（Thomson problem）的启发，超球能量最小化方法在正则化神经网络和提高其泛化能力方面显示出了巨大的潜力。本文首先通过分析超球能量的训练动力学，研究了超球能量在神经网络训练中的重要作用。然后，我们表明，天真地最小化超球能量遭受一些困难，由于高度非线性和非凸优化的空间维数变得更高，因此限制了进一步提高推广的潜力。为了解决这些问题，我们提出了压缩最小超球能量（CoMHE）作为神经网络更有效的正则化。具体而言，CoMHE利用投影映射来降低神经元的维数并最小化其超球面能量。根据投影映射的不同设计，我们提出了几个不同的但性能良好的变体，并提供了一些理论保证，以证明其有效性。我们的实验表明，CoMHE始终优于现有的正则化方法，可以很容易地应用到不同的神经网络。
英文摘要：
Inspired by the Thomson problem in physics where the distribution of multiple propelling electrons on a unit sphere can be modeled via minimizing some potential energy, hyperspherical energy minimization has demonstrated its potential in regularizing neural networks and improving their generalization power. In this paper, we first study the important role that hyperspherical energy plays in neural network training by analyzing its training dynamics. Then we show that naively minimizing hyperspherical energy suffers from some difficulties due to highly non-linear and non-convex optimization as the space dimensionality becomes higher, therefore limiting the potential to further improve the generalization. To address these problems, we propose the compressive minimum hyperspherical energy (CoMHE) as a more effective regularization for neural networks. Specifically, CoMHE utilizes projection mappings to reduce the dimensionality of neurons and minimizes their hyperspherical energy. According to different designs for the projection mapping, we propose several distinct yet well-performing variants and provide some theoretical guarantees to justify their effectiveness. Our experiments show that CoMHE consistently outperforms existing regularization methods, and can be easily applied to different neural networks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
