The remarkable reasoning and code generation capabilities of large language models (LLMs) have spurred significant interest in applying LLMs to enable task automation in digital chip design. In particular, recent work has investigated early ideas of applying these models to formal verification (FV), an approach to verifying hardware implementations that can provide strong guarantees of confidence but demands significant amounts of human effort. While the value of LLM-driven automation is evident, our understanding of model performance, however, has been hindered by the lack of holistic evaluation. In response, we present FVEval, the first comprehensive benchmark and evaluation framework for characterizing LLM performance in tasks pertaining to FV. The benchmark consists of three sub-tasks that measure LLM capabilities at different levels: from the generation of SystemVerilog assertions (SVAs) given natural language descriptions to reasoning about the design RTL and suggesting assertions directly without additional human input. As test instances, we present both collections of expert-written verification collateral and methodologies to scalably generate synthetic examples aligned with industrial FV workflows. A wide range of existing LLMs, both proprietary and open-source, are evaluated against FVEval, based on which we investigate where today's LLMs stand and how we might further enable their application toward improving productivity in digital FV. Our benchmark and evaluation code is available at \url{https://github.com/NVlabs/FVEval}.
Vulnerability Detection: From Formal Verification to Large Language Models and Hybrid Approaches: A Comprehensive Overview
Eötvös Loránd University (ELTE)(厄特沃什·罗兰大学(ELTE));Technology Innovation Institute (TII)(技术创新研究院(TII));University of Oslo(奥斯陆大学);Cyentific AS(赛恩蒂菲克公司);Guelma University(盖勒马大学);The University of Manchester(曼彻斯特大学);Federal University of Amazonas(亚马逊联邦大学)
FVEL: Interactive Formal Verification Environment with Large Language Models via Theorem Proving
Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区);City Univeristy of Hong Kong(香港城市大学);Sun Yat-sen University(中山大学);The University of Hong Kong(香港大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);MBZUAI(Mohamed bin Zayed 人工智能大学);DarkMatter AI Research(DarkMatter AI 研究院)