正则化如何影响线性网络中的临界点
How regularization affects the critical points in linear networks
摘要:
本文研究了用线性神经网络表示和学习线性变换的问题。近年来,人们对这种网络的研究越来越感兴趣,部分原因是深度学习的成功。本文的主要研究问题是均方损失函数的临界点的存在性和最优性。这里主要关注的是损失函数正则化后的临界点的鲁棒性。为此目的,引入了一个最优控制模型,并使用最优控制的Hamilton公式推导出了用于该模型的学习算法(正则化形式的反向传播)。该制剂是用来提供一个完整的表征的临界点的非线性矩阵值方程的解决方案,称为特征方程。从分歧理论的分析和数值工具被用来计算的临界点,通过特征方程的解决方案。主要结论是,临界点图可以从根本上不同,即使与任意少量的正则化。
英文摘要:
This paper is concerned with the problem of representing and learning a linear transformation using a linear neural network. In recent years, there has been a growing interest in the study of such networks in part due to the successes of deep learning. The main question of this body of research and also of this paper pertains to the existence and optimality properties of the critical points of the mean-squared loss function. The primary concern here is the robustness of the critical points with regularization of the loss function. An optimal control model is introduced for this purpose and a learning algorithm (regularized form of backprop) derived for the same using the Hamilton's formulation of optimal control. The formulation is used to provide a complete characterization of the critical points in terms of the solutions of a nonlinear matrix-valued equation, referred to as the characteristic equation. Analytical and numerical tools from bifurcation theory are used to compute the critical points via the solutions of the characteristic equation. The main conclusion is that the critical point diagram can be fundamentally different even with arbitrary small amounts of regularization.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。