Predictive coding (PC) is an energy-based learning algorithm that performs iterative inference over network activities before updating weights. Recent work suggests that PC can converge in fewer learning steps than backpropagation thanks to its inference procedure. However, these advantages are not always observed, and the impact of PC inference on learning is not theoretically well understood. Here, we study the geometry of the PC energy landscape at the inference equilibrium of the network activities. For deep linear networks, we first show that the equilibrated energy is simply a rescaled mean squared error loss with a weight-dependent rescaling. We then prove that many highly degenerate (non-strict) saddles of the loss including the origin become much easier to escape (strict) in the equilibrated energy. Our theory is validated by experiments on both linear and non-linear networks. Based on these and other results, we conjecture that all the saddles of the equilibrated energy are strict. Overall, this work suggests that PC inference makes the loss landscape more benign and robust to vanishing gradients, while also highlighting the fundamental challenge of scaling PC to deeper models.
Neural Network-based High-index Saddle Dynamics Method for Searching Saddle Points and Solution Landscape
School of Mathematical Sciences, Peking University（北京大学数学科学学院）；Center for Machine Learning Research, Peking University（机器学习研究中心）；Academy for Multidisciplinary Studies, Capital Normal University（首都师范大学交叉科学研究院）；Beijing National Center for Applied Mathematics（北京国家应用数学中心）
Symmetry, Saddle Points, and Global Optimization Landscape of Nonconvex Matrix Factorization
University of Minnesota（明尼苏达大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；Princeton University（普林斯顿大学）；Georgia Institute of Technology（佐治亚理工学院）
我们提出了一个一般性的理论来研究非凸\xl{optimization}的\xl{landscape}与潜在的对称结构\tz{for a class of machine learning problems (e.g., low-rank matrix factorization, phase retrieval, and deep linear neural networks)}。具体地，我们通过不变群\removed{for associated optimization problems, including low-rank matrix factorization, phase retrieval, and deep linear neural networks}的透镜刻画了Hessian矩阵\xl{of the objective function}的驻点位置和零空间。作为一个主要的激励例子，我们应用所提出的一般理论来表征全球\xl{landscape}的\xl{nonconvex optimization in}低秩矩阵分解问题。特别是，我们说明了如何旋转对称群产生无穷多个非孤立的严格鞍点和等价的全局极小的目标函数。通过显式识别所有静止点，我们将整个参数空间分为三个区域：（$\cR_1$）包含所有严格鞍点的邻域的区域，其中目标具有负曲率;（$\cR_2$）包含所有全局极…
Dimer-Enhanced Optimization: A First-Order Approach to Escaping Saddle Points in Neural Network Training
School of Bioengineering, Qilu University of Technology (Shandong Academy of Sciences)（青岛科技大学生物工程学院）；Shandong Provincial Key Laboratory of Biophysics, Institute of Biophysics, Dezhou University（山东省生物物理重点实验室，生物物理研究所，德州市大学）；Shandong Provincial Hospital, Shandong First Medical University（山东省人民医院，山东第一医科大学）
A Saddle Point Remedy: Power of Variable Elimination in Non-convex Optimization
College of Computer Science and Technology（计算机科学与技术学院）；College of Computer and Data Science（计算机与数据科学学院）；College of Information Engineering（信息工程学院）；School of Science（科学学院）；Shandong Provincial Key Laboratory of Network Based Intelligent Computing（网络基于智能计算的山东省重点实验室）
Riemannian ascent--descent for nonconvex nonconcave minimax landscapes: convergence to basin saddle points and applications to distributionally robust optimization
The loss landscape of deep linear neural networks: a second-order analysis
RWTH Aachen University（亚琛工业大学）；Université de Toulouse（图卢兹大学）；Institut de Mathématiques de Toulouse（图卢兹数学研究所）；CNRS（法国国家科学研究中心）；Institut de Recherche Technologique Saint Exupéry（圣埃克苏佩里技术研究院）
A Complete Loss Landscape Analysis of Regularized Deep Matrix Factorization
School of Data Science, Fudan University, Shanghai, China（复旦大学数据科学学院，上海，中国）；Department of Computer and Information Science, University of Macau, Macau SAR, China（澳门大学计算机与信息科学系，澳门特别行政区，中国）
Avoiding strict saddle points of nonconvex regularized problems
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；School of Mathematical Sciences, Shenzhen University（深圳大学数学科学学院）；Department of Applied Mathematics, The Hong Kong Polytechnic University（香港理工大学应用数学系）
A Geometric Characterization of the Stationary Plateau for Two-Layer Neural Networks
Shenzhen International Center of Industrial and Applied Mathematics（深圳工业与应用数学国际中心）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Shenzhen Loop Area Institute（深圳河套学院）；AutoKernel；University of Minnesota Twin Cities（明尼苏达大学双城分校）；School of Data Science, The Chinese University of Hong Kong, Shenzhen, China（香港中文大学（深圳）数据科学学院）
Landscaper: Understanding Loss Landscapes Through Multi-Dimensional Topological Analysis
Arizona State University（亚利桑那州立大学）；University of California Berkeley（加州大学伯克利分校）；Lawrence Berkeley National Laboratory（伯克利国家实验室）；Dartmouth College（达特茅斯学院）；International Computer Science Institute（国际计算机科学研究所）
Landscape analysis for shallow neural networks: complete classification of critical points for affine target functions
ETH Zurich（苏黎世联邦理工学院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）；University of Münster（明斯特大学）
