arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.01008cs.SEcs.AI
收藏

FVSpec: 作为精益挑战的真实世界基于属性的测试

FVSpec: Real-World Property-Based Tests as Lean Challenges

Forall R&D(Forall 研发);Benchify;Galois Inc(Galois 公司)

Quinn Dougherty, Max von Hippel, Simon Henniger, Hazel Shackleton, Mike Dodds

AI总结:

提出FVSpec基准,通过从真实Python仓库中提取属性测试并自动翻译为Lean 4规范,评估AI在形式化验证任务上的能力。

摘要:

我们提出了一个用于评估AI模型和智能体在真实世界形式化软件验证任务上的基准。首先从真实世界的Python仓库中抓取11,039个基于属性的测试(PBT),然后自动将其中2,772个(25%)翻译成9,415个带有sorry占位符的Lean 4规范(每个PBT约3个形式化;当没有形式化在质量指标上占优时,我们保留多次尝试)。将PBT翻译成Lean规范具有挑战性:需要在Lean中建模Python语义,推断命令式PBT中编码的逻辑属性,并处理在很少使用的语言中进行依赖类型编程的固有困难。我们描述了一个用于将PBT转译为Lean规范的三智能体LLM流水线,评估覆盖率和质量指标,并使用多种自动化和基于模型的方法为证明生成提供基线。所有代码(爬虫和智能体)和数据(PBT和Lean规范)都是开源的。我们的基准旨在推动AI辅助形式化验证真实世界软件这一尚未充分探索的问题的进展,随着AI生成越来越多的代码,这一问题日益受到关注。

英文摘要:

As AI systems generate an ever-growing share of the world's code, formal verification offers a principled way to ensure that code is correct, and AI itself may be able to shoulder much of the verification burden. Yet we lack a clear picture of how well today's models and agents perform on verification tasks drawn from real-world software, since existing benchmarks rely largely on curated or synthetic problems, or focus on math rather than program verification. To close this gap, we present a benchmark for evaluating AI on real-world formal software verification tasks. We first scrape 7,413 distinct property-based tests (PBTs) from real-world Python repositories, then automatically translate 2,623 of them (35%) into 9,415 Lean 4 specifications with sorry placeholders (3.5 formalizations/PBT). Translating PBTs to Lean specs is challenging: it requires modeling Python in Lean, inferring the property encoded in a PBT, and handling side effects. We describe an agentic pipeline for transpiling PBTs into Lean, evaluate coverage and quality metrics, and provide baselines for proof generation using several approaches. All code (scraper and agents) and data (PBTs and Lean specs) are open source.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑