arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.04375cs.LGcs.SYeess.SY
收藏

ECLipsE:面向深度神经网络的高效组合式Lipschitz常数估计

ECLipsE: Efficient Compositional Lipschitz Constant Estimation for Deep Neural Networks

Edwardson School of Industrial Engineering(爱德华森工业工程学院);Purdue University(普渡大学)

Yuezhu Xu, S. Sivaranjani

AI总结:

针对深度神经网络Lipschitz常数精确计算难、现有上界估计方法计算成本高的问题,提出ECLipsE组合式估计框架,通过分解问题结合两种不同精度效率权衡的算法,在保证界的紧度的同时大幅提升计算速度,增强鲁棒性验证的可扩展性。

摘要:

Lipschitz常数在证明神经网络对输入扰动的鲁棒性方面发挥着关键作用。由于计算精确的Lipschitz常数是NP难问题,已有研究致力于获取其紧上界。这类方法通常需要求解一个大型矩阵验证问题,随着网络深度和宽度的增加,计算成本会显著上升。本文提出一种组合式方法,用于估计深度前馈神经网络的Lipschitz常数。我们首先将大型矩阵验证问题精确分解为多个更小的子问题,随后利用网络的级联底层结构,开发了两种算法。第一种算法挖掘问题的几何特征,通过求解规模仅与每层神经元数相当的小型半定规划(SDPs),得到与现有方法性能相当的Lipschitz估计值。第二种算法对这些子问题进行松弛,为每个子问题提供闭式解,可实现极快的估计,完全无需求解SDPs。两种算法代表了效率与精度之间不同程度的权衡。最后,我们通过大量实验证明,所提方法能大幅缩短计算时间(针对更深层网络,根据算法不同,速度最高可达数千倍),同时得到的Lipschitz界与当前最优方法的结果非常接近,甚至更优。综上,该方法显著提升了神经网络鲁棒性验证的可扩展性与效率,尤其适用于在线学习任务。

英文摘要:

The Lipschitz constant plays a crucial role in certifying the robustness of neural networks to input perturbations. Since calculating the exact Lipschitz constant is NP-hard, efforts have been made to obtain tight upper bounds on the Lipschitz constant. Typically, this involves solving a large matrix verification problem, the computational cost of which grows significantly for both deeper and wider networks. In this paper, we provide a compositional approach to estimate Lipschitz constants for deep feed-forward neural networks. We first obtain an exact decomposition of the large matrix verification problem into smaller sub-problems. Then, leveraging the underlying cascade structure of the network, we develop two algorithms. The first algorithm explores the geometric features of the problem and enables us to provide Lipschitz estimates that are comparable to existing methods by solving small semidefinite programs (SDPs) that are only as large as the size of each layer. The second algorithm relaxes these sub-problems and provides a closed-form solution to each sub-problem for extremely fast estimation, altogether eliminating the need to solve SDPs. The two algorithms represent different levels of trade-offs between efficiency and accuracy. Finally, we demonstrate that our approach provides a steep reduction in computation time (as much as several thousand times faster, depending on the algorithm for deeper networks) while yielding Lipschitz bounds that are very close to or even better than those achieved by state-of-the-art approaches in a broad range of experiments. In summary, our approach considerably advances the scalability and efficiency of certifying neural network robustness, making it particularly attractive for online learning tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑