权重条件化以实现神经网络的平滑优化
Weight Conditioning for Smooth Optimization of Neural Networks
AI总结：
本文提出一种新的权重规范化技术，通过缩小权重矩阵的奇异值范围，提升矩阵条件数，从而改善梯度下降收敛性能，实验验证其在多种神经网络架构中的有效性。
摘要：
在本文中，我们介绍了一种新的神经网络权重矩阵规范化技术，称为权重条件化。该方法旨在缩小权重矩阵的最小和最大奇异值之间的差距，从而得到更良好的条件矩阵。该技术的灵感部分来源于数值线性代数，其中良好的条件矩阵已知能为迭代求解器提供更强的收敛结果。我们提供了理论基础，证明我们的规范化技术平滑了损失景观，从而增强随机梯度下降算法的收敛性。实证上，我们在各种神经网络架构中验证了我们的规范化方法，包括卷积神经网络（CNN）、视觉Transformer（ViT）、神经辐射场（NeRF）和3D形状建模。我们的发现表明，我们的规范化方法不仅具有竞争力，而且在文献中现有的权重规范化技术上也表现更优。
英文摘要：
In this article, we introduce a novel normalization technique for neural network weight matrices, which we term weight conditioning. This approach aims to narrow the gap between the smallest and largest singular values of the weight matrices, resulting in better-conditioned matrices. The inspiration for this technique partially derives from numerical linear algebra, where well-conditioned matrices are known to facilitate stronger convergence results for iterative solvers. We provide a theoretical foundation demonstrating that our normalization technique smoothens the loss landscape, thereby enhancing convergence of stochastic gradient descent algorithms. Empirically, we validate our normalization across various neural network architectures, including Convolutional Neural Networks (CNNs), Vision Transformers (ViT), Neural Radiance Fields (NeRF), and 3D shape modeling. Our findings indicate that our normalization method is not only competitive but also outperforms existing weight normalization techniques from the literature.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
