s2n-bignum-bench:评估LLM低级代码推理能力的实用基准
s2n-bignum-bench: A practical benchmark for evaluating low-level code reasoning of LLMs
AI总结:
本文提出s2n-bignum-bench基准,用于评估LLM在工业级低级密码学汇编代码中的定理证明能力,通过形式化验证挑战证明生成。
摘要:
利用大型语言模型(LLMs)与形式方法的神经符号方法最近在数学导向的定理证明基准上取得了强劲成果。然而,竞赛式数学的成功本身并不表明能够构造关于现实世界实现的证明。我们通过一个源自工业加密库的基准来填补这一空白,该库的汇编程序已在HOL Light中已验证。s2n-bignum是AWS用于提供快速加密汇编程序的库,其正确性通过形式验证建立。在s2n-bignum-bench中,我们提供形式化规范,并要求LLM生成一个在固定证明检查超时内被HOL Light接受的证明脚本。据我们所知,s2n-bignum-bench是首个专注于HOL Light中工业级低级密码学汇编程序机器可检查证明合成的公开基准。此基准为评估LLM基于定理证明超越竞赛数学提供了具有挑战性和实际相关性的测试环境。设置和使用此基准的代码在此处可用:https://github.com/kings-crown/s2n-bignum-bench。
英文摘要:
Neurosymbolic approaches leveraging Large Language Models (LLMs) with formal methods have recently achieved strong results on mathematics-oriented theorem-proving benchmarks. However, success on competition-style mathematics does not by itself demonstrate the ability to construct proofs about real-world implementations. We address this gap with a benchmark derived from an industrial cryptographic library whose assembly routines are already verified in HOL Light. s2n-bignum is a library used at AWS for providing fast assembly routines for cryptography, and its correctness is established by formal verification. The task of formally verifying this library has been a significant achievement for the Automated Reasoning Group. It involved two tasks: (1) precisely specifying the correct behavior of a program as a mathematical proposition, and (2) proving that the proposition is correct. In the case of s2n-bignum, both tasks were carried out by human experts. In \textit{s2n-bignum-bench}, we provide the formal specification and ask the LLM to generate a proof script that is accepted by HOL Light within a fixed proof-check timeout. To our knowledge, \textit{s2n-bignum-bench} is the first public benchmark focused on machine-checkable proof synthesis for industrial low-level cryptographic assembly routines in HOL Light. This benchmark provides a challenging and practically relevant testbed for evaluating LLM-based theorem proving beyond competition mathematics. The code to set up and use the benchmark is available here: \href{https://github.com/kings-crown/s2n-bignum-bench}{s2n-bignum-bench}.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。