神经网络光滑性的定量几何方法
A Quantitative Geometric Approach to Neural-Network Smoothness
摘要：
快速精确地估计神经网络的Lipschitz常数是深度学习的重要任务。研究人员最近发现，神经网络的准确性和平滑性之间存在内在的权衡，因此使用松散的Lipschitz常数估计来训练网络会带来很强的正则化，并且会显着损害模型的准确性。在这项工作中，我们提供了一个统一的理论框架，一个定量的几何方法，来解决Lipschitz常数估计。通过采用这个框架，我们可以立即得到几个理论结果，包括Lipschitz常数估计的计算难度和它的逼近性。此外，定量几何的角度也可以提供一些见解，最近的经验观察，一个规范的技术通常不会转移到另一个。我们还实现了从这种定量几何方法在一个工具GeoLIP的算法。这些算法是基于半定规划（SDP）。我们的实证评估表明，GeoLIP是更可扩展的和精确的比现有的工具Lipschitz常数估计$\ell_\infty$扰动。此外，我们还展示了它与其他最近的基于SDP的技术，无论是理论上还是经验上的错综复杂的关系。我们相信，这种统一的定量几何视角可以为神经网络的平滑性和鲁棒性的研究带来新的见解和理论工具。
英文摘要：
Fast and precise Lipschitz constant estimation of neural networks is an important task for deep learning. Researchers have recently found an intrinsic trade-off between the accuracy and smoothness of neural networks, so training a network with a loose Lipschitz constant estimation imposes a strong regularization and can hurt the model accuracy significantly. In this work, we provide a unified theoretical framework, a quantitative geometric approach, to address the Lipschitz constant estimation. By adopting this framework, we can immediately obtain several theoretical results, including the computational hardness of Lipschitz constant estimation and its approximability. Furthermore, the quantitative geometric perspective can also provide some insights into recent empirical observations that techniques for one norm do not usually transfer to another one. We also implement the algorithms induced from this quantitative geometric approach in a tool GeoLIP. These algorithms are based on semidefinite programming (SDP). Our empirical evaluation demonstrates that GeoLIP is more scalable and precise than existing tools on Lipschitz constant estimation for $\ell_\infty$-perturbations. Furthermore, we also show its intricate relations with other recent SDP-based techniques, both theoretically and empirically. We believe that this unified quantitative geometric perspective can bring new insights and theoretical tools to the investigation of neural-network smoothness and robustness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
