arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.23299cs.ARcs.AI
收藏

FVEval:理解语言模型在数字硬件形式验证中的能力

FVEval: Understanding Language Model Capabilities in Formal Verification of Digital Hardware

UC Berkeley(加州大学伯克利分校);NVIDIA(英伟达公司)

Minwoo Kang, Mingjie Liu, Ghaith Bany Hamad, Syed Suhaib, Haoxing Ren

AI总结:

针对大语言模型在数字硬件形式验证领域缺乏全面性能评估的问题,研究者提出首个综合基准FVEval,覆盖三类不同难度的验证子任务,评估了多款专有与开源LLM的表现,为推动其在工业形式验证中提升效率提供参考。

摘要:

大语言模型(LLM)卓越的推理和代码生成能力,激发了人们将LLM应用于数字芯片设计任务自动化的浓厚兴趣。特别是近期研究探索了将这类模型应用于形式验证(FV)的初步思路——形式验证是一种验证硬件实现的方法,可提供强可信度保证,但需要投入大量人力。尽管LLM驱动的自动化价值显著,但由于缺乏全面评估,我们对模型性能的认知仍受限制。为此,我们提出FVEval,这是首个用于表征LLM在FV相关任务中性能的综合基准与评估框架。该基准包含三个子任务,从不同层面衡量LLM能力:从根据自然语言描述生成SystemVerilog断言(SVA),到对设计RTL进行推理并在无需额外人工输入的情况下直接给出断言建议。作为测试用例,我们既提供了专家编写的验证辅助资料集合,也提出了可规模化生成符合工业FV工作流的合成示例的方法。我们基于FVEval评估了大量现有LLM,包括专有模型和开源模型,借此探究当前LLM的水平,以及如何进一步推动其应用以提升数字FV的生产效率。我们的基准和评估代码可在https://github.com/NVlabs/FVEval获取。

英文摘要:

The remarkable reasoning and code generation capabilities of large language models (LLMs) have spurred significant interest in applying LLMs to enable task automation in digital chip design. In particular, recent work has investigated early ideas of applying these models to formal verification (FV), an approach to verifying hardware implementations that can provide strong guarantees of confidence but demands significant amounts of human effort. While the value of LLM-driven automation is evident, our understanding of model performance, however, has been hindered by the lack of holistic evaluation. In response, we present FVEval, the first comprehensive benchmark and evaluation framework for characterizing LLM performance in tasks pertaining to FV. The benchmark consists of three sub-tasks that measure LLM capabilities at different levels: from the generation of SystemVerilog assertions (SVAs) given natural language descriptions to reasoning about the design RTL and suggesting assertions directly without additional human input. As test instances, we present both collections of expert-written verification collateral and methodologies to scalably generate synthetic examples aligned with industrial FV workflows. A wide range of existing LLMs, both proprietary and open-source, are evaluated against FVEval, based on which we investigate where today's LLMs stand and how we might further enable their application toward improving productivity in digital FV. Our benchmark and evaluation code is available at \url{https://github.com/NVlabs/FVEval}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑