arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.28113cs.LG
收藏

解开Lipschitz验证之谜:正矩阵、负结果

The multiplicative Lipschitz bound as an inductive bias

Department of Mechanical and Aerospace Engineering(机械与航空航天工程系);University of California, Davis(加州大学戴维斯分校);Edwardson School of Industrial Engineering(工业工程学院);Purdue University(普渡大学)

Simon Kuang, Yuezhu Xu, S. Sivaranjani, Xinfan Lin

AI总结:

本文探讨了神经网络全局Lipschitz常数的估计难题,指出其结构性困难源于可达性问题的NP难性,并通过实例证明SDP方法的局限性,同时提出通过优化trivial bound和引入三角形层可使验证结果更精确。

摘要:

神经网络的全局Lipschitz常数与鲁棒性和泛化能力相关,但不同于经典模型,它不能直接从参数中看出。这推动了复杂的验证算法,尤其是基于半正定规划(SDP)的增量二次约束方法,以改进快速但通常松散的逐层Lipschitz常数乘积(平凡界)。本文探讨了为何Lipschitz验证本身是个问题,指出困难是结构性的:估计网络的Lipschitz常数需要知道哪些隐藏状态可达,而可达性是NP难的。如果P≠NP,则可达性成为任何多项式时间算法的障碍。通过显式构造,我们证明这种盲区可使SDP界继承与平凡界相同的定性失败,包括但不限于多项式每层保守性。我们展示NP难问题的困难不仅局限于最坏情况的计算减少,而是实际上影响验证问题的每一个实例。因此SDP不足以进行Lipschitz验证。我们还论证这并非必要:一些看似平凡界的失败源于可去除的参数化病理现象,可通过优化或正则化平凡界本身来缓解。我们通过一个“球状牛”线性模型和数值证明概念来展示这一主张。虽然主要贡献是理论性的和负面的,我们最后提出了一种新的三角形层形式,它们不需要偏置即可实现通用逼近。结合平凡界正则化,它们使平凡界在理论上和实践中都严格成立。

英文摘要:

We study the problem of training a neural network to minimize a loss function given a Lipschitz constraint. While the most obvious way to estimate the Lipschitz constant of a neural network is to multiply the Lipschitz constants of its layers, this bound, which we term the _trivial bound_, is often considered impractically loose. We show that, on the contrary, it can be more effective than state-of-the-art methods (Sandwich, Almost-Orthogonal Layers, Convex Potential Layers, and SDP-based Lipschitz Layers) for Lipschitz-constrained learning. Our method combines the trivial bound with a novel activation function (the quadrature nonlinearity) to achieve lower cross-entropy over most of the tested classification bounds. For autoencoders constrained to be nonexpansive in a learned norm, it achieves the lowest pixel reconstruction error at every tested bottleneck width, improving on PCA by 12--24%. Our architecture, consisting of concatenated cosine and sine activations, is supported by a semi-rigorous theoretical analysis of Lipschitz bound looseness that may prove valuable in its own right.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑