arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2102.13233cs.LG
收藏

伪局部最小值在具有分段线性激活的深度神经网络中很常见

Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activations

College of Computer Science, Faculty of Information Technology(信息技术学院计算机学院);Beijing University of Technology(北京工业大学)

Bo Liu

摘要:

在本文中,理论上证明了伪局部极小值对于具有分段线性激活函数和无法用线性模型拟合的数据集的深度全连接网络和卷积神经网络(CNN)是常见的。给出了一个激励性的例子来解释伪局部最小值存在的原因:深度全连接网络和具有分段线性激活的CNN的每个输出神经元产生连续分段线性(CPWL)输出,并且不同的CPWL输出可以在最小化经验风险时拟合不相交的数据样本组。用不同的CPWL函数拟合数据样本通常会导致不同程度的经验风险,从而导致伪局部极小值的流行。这个结果在一般情况下证明了任何连续的损失函数。主要的证明技术是将CPWL函数表示为线性片段的最小化上的最大化。然后构建深度ReLU网络来产生这些线性片段,并实现最大化和最小化操作。

英文摘要:

In this paper, it is shown theoretically that spurious local minima are common for deep fully-connected networks and convolutional neural networks (CNNs) with piecewise linear activation functions and datasets that cannot be fitted by linear models. A motivating example is given to explain the reason for the existence of spurious local minima: each output neuron of deep fully-connected networks and CNNs with piecewise linear activations produces a continuous piecewise linear (CPWL) output, and different pieces of CPWL output can fit disjoint groups of data samples when minimizing the empirical risk. Fitting data samples with different CPWL functions usually results in different levels of empirical risk, leading to prevalence of spurious local minima. This result is proved in general settings with any continuous loss function. The main proof technique is to represent a CPWL function as a maximization over minimization of linear pieces. Deep ReLU networks are then constructed to produce these linear pieces and implement maximization and minimization operations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑