arXiv 2201.121982023-09-07cs.LG

用数据无关函数刻画隐式正则化的局限性

Limitation of Characterizing Implicit Regularization by Data-independent Functions

University of Illinois - Urbana Champaign（伊利诺伊大学厄巴纳-香槟分校）；Shanghai Jiao Tong University（上海交通大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Center for Brain Science and Brain-Inspired Technology（上海脑科学与类脑研究中心）

Leyang Zhang, Zhi-Qin John Xu, Tao Luo, Yaoyu Zhang

备注：: Revised the structure of paper and added results about implicit regularization in training two-layer network or even more general "activation function-related" models

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文从数学上定义并研究了神经网络隐式正则化，提出两点和单点重叠机制，证明了使用数据无关函数刻画隐式正则化的局限性，强调了其深刻的数据依赖性。

摘要：

近年来，理解神经网络（NNs）的隐式正则化已成为深度学习理论的核心任务。然而，隐式正则化本身尚未被完全定义和理解。在本工作中，我们尝试从数学上定义并研究隐式正则化。重要的是，我们探讨了使用数据无关函数刻画隐式正则化的常见方法的局限性。我们提出了两种动力学机制，即两点重叠机制和单点重叠机制，基于此我们提供了两种构建单隐藏神经元NNs类的方法，可证明这些网络无法被某类或所有数据无关函数完全刻画。继先前的研究之后，我们的结果进一步强调了隐式正则化普遍存在的深刻数据依赖性，启发我们未来深入研究NN隐式正则化的数据依赖性。

英文摘要：

In recent years, understanding the implicit regularization of neural networks (NNs) has become a central task in deep learning theory. However, implicit regularization is itself not completely defined and well understood. In this work, we attempt to mathematically define and study implicit regularization. Importantly, we explore the limitations of a common approach to characterizing implicit regularization using data-independent functions. We propose two dynamical mechanisms, i.e., Two-point and One-point Overlapping mechanisms, based on which we provide two recipes for producing classes of one-hidden-neuron NNs that provably cannot be fully characterized by a type of or all data-independent functions. Following the previous works, our results further emphasize the profound data dependency of implicit regularization in general, inspiring us to study in detail the data dependency of NN implicit regularization in the future.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。