arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.11053cs.ARcs.AI
收藏

重访 VerilogEval:大语言模型在硬件代码生成领域一年来的进展

Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generation

NVIDIA Corporation(英伟达公司);Cornell University(康奈尔大学)

Nathaniel Pinckney, Christopher Batten, Mingjie Liu, Haoxing Ren, Brucek Khailany

AI总结:

本文重新评估了 VerilogEval 发布一年来多种新商业与开源大语言模型在硬件代码生成上的表现,发现 GPT-4o 与 Llama3.1 405B 等模型通过率显著提升,并通过改进基准测试基础设施强调了提示工程与失败分析对模型发展的重要性。

摘要:

大语言模型(LLMs)在数字硬件代码生成中的应用是一个新兴领域,大多数 LLM 主要基于自然语言和软件代码进行训练。像 Verilog 这样的硬件代码只占训练数据的一小部分,且现有的硬件基准测试很少。2023 年 11 月发布的开源 VerilogEval 基准测试为 LLM 在代码补全任务上提供了一致的评估框架。自那时以来,商业模型和开源模型都取得了显著发展。在这项工作中,我们针对自 VerilogEval 最初发布以来的新商业和开源模型——包括 GPT-4o、GPT-4 Turbo、Llama3.1(8B/70B/405B)、Llama3 70B、Mistral Large、DeepSeek Coder(33B 和 6.7B)、CodeGemma 7B 以及 RTL-Coder——在改进后的 VerilogEval 基准测试套件上进行评估。我们发现最先进模型取得了可衡量的改进:GPT-4o 在 specification-to-RTL 任务上达到了 63% 的通过率。最近发布的开源 Llama3.1 405B 达到了 58% 的通过率,几乎与 GPT-4o 持平,而规模较小的领域专用 RTL-Coder 6.7B 模型则取得了令人印象深刻的 34% 通过率。此外,我们通过自动分类失败原因、引入上下文学习支持以及将任务扩展到 specification-to-RTL 翻译,增强了 VerilogEval 的基础设施。我们发现提示工程对于获得良好的通过率仍然至关重要,并且其效果随模型和任务的不同而有很大差异。一个允许进行提示工程和失败分析的基准测试基础设施对于持续的模型开发和部署至关重要。

英文摘要:

The application of large-language models (LLMs) to digital hardware code generation is an emerging field, with most LLMs primarily trained on natural language and software code. Hardware code like Verilog constitutes a small portion of training data, and few hardware benchmarks exist. The open-source VerilogEval benchmark, released in November 2023, provided a consistent evaluation framework for LLMs on code completion tasks. Since then, both commercial and open models have seen significant development. In this work, we evaluate new commercial and open models since VerilogEval's original release-including GPT-4o, GPT-4 Turbo, Llama3.1 (8B/70B/405B), Llama3 70B, Mistral Large, DeepSeek Coder (33B and 6.7B), CodeGemma 7B, and RTL-Coder-against an improved VerilogEval benchmark suite. We find measurable improvements in state-of-the-art models: GPT-4o achieves a 63% pass rate on specification-to-RTL tasks. The recently released and open Llama3.1 405B achieves a 58% pass rate, almost matching GPT-4o, while the smaller domain-specific RTL-Coder 6.7B models achieve an impressive 34% pass rate. Additionally, we enhance VerilogEval's infrastructure by automatically classifying failures, introducing in-context learning support, and extending the tasks to specification-to-RTL translation. We find that prompt engineering remains crucial for achieving good pass rates and varies widely with model and task. A benchmark infrastructure that allows for prompt engineering and failure analysis is essential for continued model development and deployment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑