arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.21063cs.LGcs.AI
收藏

面向一般非线性函数的神经网络分支定界验证

Neural Network Verification with Branch-and-Bound for General Nonlinearities

University of California, Los Angeles(加州大学洛杉矶分校);University of Michigan(密歇根大学);Carnegie Mellon University(卡内基梅隆大学);Columbia University(哥伦比亚大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)

Zhouxing Shi, Qirui Jin, Zico Kolter, Suman Jana, Cho-Jui Hsieh, Huan Zhang

AI总结:

针对现有神经网络验证分支定界主要局限于ReLU等分段线性激活的问题,本文提出通用框架GenBaB,通过线性边界传播、新型分支启发式与预优化分支点查找表,支持Sigmoid、Tanh、Sine、GeLU及LSTM/ViT中的多维非线性运算,并能验证ACOPF等一般非线性计算图。

摘要:

分支定界(BaB)是神经网络(NN)验证中最有效的技术之一。然而,现有的神经网络验证BaB研究大多集中于具有分段线性激活函数的网络,尤其是ReLU网络。本文基于用于神经网络验证的线性边界传播方法,开发了一个名为GenBaB的通用框架,以对一般非线性函数执行BaB,从而验证具有通用架构的神经网络。为了决定对哪个神经元进行分支,我们设计了一种新的分支启发式方法,该方法利用线性边界作为捷径,高效估计分支后可能带来的改进。为了确定一般非线性函数的非平凡分支点,我们提出预先优化分支点,并在验证过程中通过查找表高效利用这些分支点。我们展示了GenBaB在验证多种神经网络方面的有效性,包括具有Sigmoid、Tanh、Sine和GeLU等激活函数的神经网络,以及涉及多维非线性运算的网络,例如LSTM和Vision Transformers中的乘法运算。我们的框架还支持验证一般非线性计算图,并使得超越简单神经网络的验证应用成为可能,特别是针对交流最优潮流(ACOPF)问题。GenBaB是最新版本α,β-CROWN的一部分,该工具是第四届和第五届国际神经网络验证竞赛(VNN-COMP 2023和2024)的获胜者。用于复现实验的代码可在https://github.com/shizhouxing/GenBaB获取。

英文摘要:

Branch-and-bound (BaB) is among the most effective techniques for neural network (NN) verification. However, existing works on BaB for NN verification have mostly focused on NNs with piecewise linear activations, especially ReLU networks. In this paper, we develop a general framework, named GenBaB, to conduct BaB on general nonlinearities to verify NNs with general architectures, based on linear bound propagation for NN verification. To decide which neuron to branch, we design a new branching heuristic which leverages linear bounds as shortcuts to efficiently estimate the potential improvement after branching. To decide nontrivial branching points for general nonlinear functions, we propose to pre-optimize branching points, which can be efficiently leveraged during verification with a lookup table. We demonstrate the effectiveness of our GenBaB on verifying a wide range of NNs, including NNs with activation functions such as Sigmoid, Tanh, Sine and GeLU, as well as NNs involving multi-dimensional nonlinear operations such as multiplications in LSTMs and Vision Transformers. Our framework also allows the verification of general nonlinear computation graphs and enables verification applications beyond simple NNs, particularly for AC Optimal Power Flow (ACOPF). GenBaB is part of the latest $α$,$β$-CROWN, the winner of the 4th and the 5th International Verification of Neural Networks Competition (VNN-COMP 2023 and 2024). Code for reproducing the experiments is available at https://github.com/shizhouxing/GenBaB.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑