The application of large-language models (LLMs) to digital hardware code generation is an emerging field, with most LLMs primarily trained on natural language and software code. Hardware code like Verilog constitutes a small portion of training data, and few hardware benchmarks exist. The open-source VerilogEval benchmark, released in November 2023, provided a consistent evaluation framework for LLMs on code completion tasks. Since then, both commercial and open models have seen significant development.
In this work, we evaluate new commercial and open models since VerilogEval's original release-including GPT-4o, GPT-4 Turbo, Llama3.1 (8B/70B/405B), Llama3 70B, Mistral Large, DeepSeek Coder (33B and 6.7B), CodeGemma 7B, and RTL-Coder-against an improved VerilogEval benchmark suite. We find measurable improvements in state-of-the-art models: GPT-4o achieves a 63% pass rate on specification-to-RTL tasks. The recently released and open Llama3.1 405B achieves a 58% pass rate, almost matching GPT-4o, while the smaller domain-specific RTL-Coder 6.7B models achieve an impressive 34% pass rate.
Additionally, we enhance VerilogEval's infrastructure by automatically classifying failures, introducing in-context learning support, and extending the tasks to specification-to-RTL translation. We find that prompt engineering remains crucial for achieving good pass rates and varies widely with model and task. A benchmark infrastructure that allows for prompt engineering and failure analysis is essential for continued model development and deployment.
EvoVerilog: Large Langugage Model Assisted Evolution of Verilog Code
Department of Computer Science, City University of Hong Kong, Hong Kong(香港城市大学计算机科学系);Department of Electrical Engineering, University of Maryland, Maryland, United States(马里兰大学电气工程系);Amazon, Seattle, United States(亚马逊公司)
ReasoningV: Efficient Verilog Code Generation with Adaptive Hybrid Reasoning Model
National Key Laboratory of Spintronics, Hangzhou International Innovation Institute(国家磁电子重点实验室、杭州国际创新研究院);School of Integrated Circuit Science and Engineering, Beihang University(集成电路科学与工程学院、北京航空航天大学)
CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations and Targeted Code Repair
EvolVE: Evolutionary Search for LLM-based Verilog Generation and Optimization
Department of Electrical Engineering(电气工程系);National Taiwan University(国立台湾大学);Department of Computer Science and Information Engineering(计算机科学与信息工程系);Institute of Information Science(信息科学研究所)
CodeV: Empowering LLMs with HDL Generation through Multi-Level Summarization
State Key Lab of Processors, Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所处理器重点实验室);University of Chinese Academy of Sciences(中国科学院大学);Cambricon Technologies(寒武科技);School of Information Science and Technology, ShanghaiTech University(上海科技大学信息科学与技术学院);University of Science and Technology of China(中国科学技术大学);Shanghai Innovation Center for Processor Technologies(上海处理器技术创新中心)
QiMeng-SALV: Signal-Aware Learning for Verilog Code Generation
State Key Lab of Processors, Institute of Computing Technology, CAS(处理器国家重点实验室,计算技术研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学);University of Science and Technology of China(中国科学技术大学)