arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.03154cs.LGcs.AIcs.SE
收藏

SoundnessBench: 一种用于神经网络验证器的声学基准

SoundnessBench: A Soundness Benchmark for Neural Network Verifiers

University of California, Los Angeles(加州大学洛杉矶分校);Georgia Institute of Technology(佐治亚理工学院);Duke University(杜克大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of California, Riverside(加州大学河滨分校)

Xingjian Zhou, Keyi Shen, Andy Xu, Hongji Xu, Cho-Jui Hsieh, Huan Zhang, Zhouxing Shi

AI总结:

SoundnessBench通过构建包含隐藏反例的基准,用于评估神经网络验证器的正确性。

摘要:

神经网络(NN)验证旨在正式验证NN的属性,这对于确保NN模型在安全关键应用中的行为至关重要。近年来,社区已开发了许多NN验证器和基准,用于评估它们。然而,现有基准通常缺乏硬实例的地面真实值,这些实例中没有任何当前验证器可以验证属性,也没有反例可以找到。这使得在声称对这些具有挑战性的实例进行验证时,验证验证器的正确性变得困难。在本工作中,我们开发了一个新的NN验证基准,名为SoundnessBench,专门用于测试NN验证器的正确性。SoundnessBench包含故意插入的反例,这些反例被隐藏在常见的对抗攻击中,用于寻找反例。因此,它可以识别出当已知存在隐藏反例时的虚假验证声明。我们设计了一种训练方法来生成具有隐藏反例的NN,并系统地构建我们的SoundnessBench,涵盖各种模型架构、激活函数和输入数据。我们证明了我们的训练能够有效生成隐藏反例,我们的SoundnessBench成功识别了最先进的NN验证器中的bug。我们的代码可在https://github.com/mvp-harry/SoundnessBench上获得,我们的数据集可在https://huggingface.co/datasets/SoundnessBench/SoundnessBench上获得。

英文摘要:

Neural network (NN) verification aims to formally verify properties of NNs, which is crucial for ensuring the behavior of NN-based models in safety-critical applications. In recent years, the community has developed many NN verifiers and benchmarks to evaluate them. However, existing benchmarks typically lack ground-truth for hard instances where no current verifier can verify the property and no counterexample can be found. This makes it difficult to validate the soundness of a verifier, when it claims verification on such challenging instances that no other verifier can handle. In this work, we develop a new benchmark for NN verification, named SoundnessBench, specifically for testing the soundness of NN verifiers. SoundnessBench consists of instances with deliberately inserted counterexamples that are hidden from adversarial attacks commonly used to find counterexamples. Thereby, it can identify false verification claims when hidden counterexamples are known to exist. We design a training method to produce NNs with hidden counterexamples and systematically construct our SoundnessBench with instances across various model architectures, activation functions, and input data. We demonstrate that our training effectively produces hidden counterexamples and our SoundnessBench successfully identifies bugs in state-of-the-art NN verifiers. Our code is available at https://github.com/mvp-harry/SoundnessBench and our dataset is available at https://huggingface.co/datasets/SoundnessBench/SoundnessBench.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑