用可证明稳健实例压力测试神经网络验证器
Stress-Testing Neural Network Verifiers with Provably Robust Instances
AI总结:
本文提出了一种生成具有已知真实稳健标签的验证实例的框架,揭示了现有验证器的数值容忍度问题和实现错误,并引入了验证难度轮廓以系统研究验证器失败模式,评估了五种最先进的验证器并展示了不同实例对验证流程不同方面的压力测试。
摘要:
神经网络验证器旨在为模型行为提供正式保证,但现有的验证基准本质上受到缺乏真实标签的限制。因此,验证器评估依赖于间接启发式方法,这阻止了精确评分和系统研究验证器失败模式。我们通过引入一个可重用的框架来生成验证实例,其真实稳健标签通过分析构造已知,从而填补了这一差距。我们的框架导致在流行的验证器中发现了多个数值容忍度问题和实现错误,突显了真实标签的必要性。此外,为了系统研究验证器失败模式,我们引入了验证难度轮廓,一个收集可估计数量的集合,捕捉不同的实例难度来源。使用我们的框架和这些轮廓,我们评估了五种最先进的验证器,并展示了不同实例对验证流程不同方面的压力测试。我们证明这些结果可以帮助未来验证器的发展,因为它们为提高数值可靠性、放松质量和搜索行为提供了可行的目标。我们的代码已公开可用:https://github.com/dtroxell19/VeriStressGT.git。
英文摘要:
Neural network verifiers aim to provide formal guarantees on model behavior, but existing verification benchmarks are fundamentally limited by their lack of ground-truth labels. As a result, verifier evaluation relies on indirect heuristics, which prevents exact scoring and systematic study of verifier failure modes. We address this gap by introducing a reusable framework for generating verification instances whose ground-truth robustness labels are known a priori through analytic construction. Our framework led to the discovery of multiple numeric tolerance concerns and an implementation bug in popular verifiers, highlighting the need for ground-truth labels. Additionally, to systematically study verifier failure modes, we introduce the verification Difficulty Profile, a collection of estimable quantities capturing distinct sources of instance hardness. Using our framework and these profiles, we evaluate five state-of-the-art verifiers and show that different instances stress distinct aspects of the verification pipeline. We show that these results can aid the future development of verifiers as they provide actionable targets for improving numerical reliability, relaxation quality, and search behavior. Our code is publicly available: https://github.com/dtroxell19/VeriStressGT.git.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。