惩罚范数正则化中的偏置项可强制稀疏性
Penalising the biases in norm regularisation enforces sparsity
AI总结：
本文针对一维数据单隐层ReLU网络，揭示了参数范数正则化中惩罚偏置项会引入额外加权因子，从而强制最小范数插值器具有唯一性和稀疏性。
摘要：
在训练神经网络时，控制参数的范数通常能带来良好的泛化性能。然而，除了简单的直觉之外，参数范数正则化与所得到的估计器之间的关系在理论上仍未得到充分理解。针对具有一维数据的单隐层ReLU网络，本工作表明，表示一个函数所需的参数范数由其二阶导数的总变分给出，并以一个 $\sqrt{1+x^2}$ 因子加权。值得注意的是，当偏置项的范数未被正则化时，该加权因子会消失。这一额外加权因子的存在具有极其重要的意义，因为研究表明它能够强制最小范数插值器的唯一性和稀疏性（以折点数量衡量）。相反，省略偏置项的范数则允许非稀疏解的存在。因此，在正则化中显式或隐式地惩罚偏置项，可以得到稀疏的估计器。
英文摘要：
Controlling the parameters' norm often yields good generalisation when training neural networks. Beyond simple intuitions, the relation between regularising parameters' norm and obtained estimators remains theoretically misunderstood. For one hidden ReLU layer networks with unidimensional data, this work shows the parameters' norm required to represent a function is given by the total variation of its second derivative, weighted by a $\sqrt{1+x^2}$ factor. Notably, this weighting factor disappears when the norm of bias terms is not regularised. The presence of this additional weighting factor is of utmost significance as it is shown to enforce the uniqueness and sparsity (in the number of kinks) of the minimal norm interpolator. Conversely, omitting the bias' norm allows for non-sparse solutions. Penalising the bias terms in the regularisation, either explicitly or implicitly, thus leads to sparse estimators.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
