Charged Point Normalization: An Efficient Solution to the Saddle Point Problem

The Power of Normalization: Faster Evasion of Saddle Points

A commonly used heuristic in non-convex optimization is Normalized Gradient Descent (NGD) - a variant of gradient descent in which only the direction of the gradient is taken into account and its magnitude ignored. We analyze this heuristic and show that with carefully chosen parameters and noise injection, this method can provably evade saddle points. We establish the convergence of NGD to a local minimum, and demonstrate rates which improve upon the fastest known first order algorithm due to Ge e al. (2015). The effectiveness of our method is demonstrated via an application to the problem of…

Vanishing Curvature and the Power of Adaptive Methods in Randomly Initialized Deep Networks

This paper revisits the so-called vanishing gradient phenomenon, which commonly occurs in deep randomly initialized neural networks. Leveraging an in-depth analysis of neural chains, we first show that vanishing gradients cannot be circumvented when the network width scales with less than O(depth), even when initialized with the popular Xavier and He initializations. Second, we extend the analysis to second-order derivatives and show that random i.i.d. initialization also gives rise to Hessian matrices with eigenspectra that vanish as networks grow in depth. Whenever this happens, optimizers a…

等变性打破学习率

Equivariance Breaks the Learning Rate

本文发现等变网络中不可约表示块的梯度结构导致Adam学习率不匹配，提出块归一化方法，结合动量调整使Adam在势模型上媲美Muon。

使用原始-对偶 Langevin Monte Carlo 的约束采样

Constrained Sampling with Primal-Dual Langevin Monte Carlo

针对带非线性期望统计约束的分布采样问题，提出离散时间原始-对偶 PD-LMC 算法，并在凸性与对数 Sobolev 不等式假设下证明其收敛性。

深度嵌入原理在深度神经网络损失景观分析中的应用

Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networks

该研究发现深度嵌入原理，即深层神经网络损失景观包含浅层所有临界点，通过临界提升算子映射临界点并保留输出，为深度网络易训练性、批量归一化加速等行为提供新解释，揭示损失景观深度层次结构。

Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold

When training overparameterized deep networks for classification tasks, it has been widely observed that the learned features exhibit a so-called "neural collapse" phenomenon. More specifically, for the output features of the penultimate layer, for each class the within-class features converge to their means, and the means of different classes exhibit a certain tight frame structure, which is also aligned with the last layer's classifier. As feature normalization in the last layer becomes a common practice in modern representation learning, in this work we theoretically justify the neural coll…