消解神经网络优化中的奇异性诅咒
Dispelling the Curse of Singularities in Neural Network Optimizations
AI总结：
本文提出PSS方法，通过平滑权重矩阵的奇异谱来缓解神经网络优化中的奇异性诅咒，提升训练稳定性和泛化能力。
摘要：
本文从一个较少被探索但具有洞察力的视角研究深度神经网络优化不稳定性的根源：参数空间中奇异性的出现与放大。我们的分析揭示，参数奇异性不可避免地随着梯度更新而增长，并进一步加剧与表征的对齐，导致表征空间中的奇异性增加。我们证明梯度Frobenius范数被权重矩阵的上奇异值所限制，并且随着训练进行，权重和表征奇异性相互强化的增长，称为奇异性诅咒，会放松这些界限，增加损失爆炸的风险。为应对这一问题，我们提出了参数奇异性平滑（PSS），一种轻量、灵活且有效的平滑权重矩阵奇异谱的方法。在多样化的数据集、架构和优化器上进行的广泛实验表明，PSS减轻了不稳定性，即使在失败后也能恢复可训练性，并提高了训练效率和泛化能力。
英文摘要：
This work investigates the optimization instability of deep neural networks from a less-explored yet insightful perspective: the emergence and amplification of singularities in the parametric space. Our analysis reveals that parametric singularities inevitably grow with gradient updates and further intensify alignment with representations, leading to increased singularities in the representation space. We show that the gradient Frobenius norms are bounded by the top singular values of the weight matrices, and as training progresses, the mutually reinforcing growth of weight and representation singularities, termed the curse of singularities, relaxes these bounds, escalating the risk of sharp loss explosions. To counter this, we propose Parametric Singularity Smoothing (PSS), a lightweight, flexible, and effective method for smoothing the singular spectra of weight matrices. Extensive experiments across diverse datasets, architectures, and optimizers demonstrate that PSS mitigates instability, restores trainability even after failure, and improves both training efficiency and generalization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
