arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.11287cs.ARcs.SE
收藏

LLM在RTL生成中的回路合成评估:质量、可靠性与故障模式

Synthesis-in-the-Loop Evaluation of LLMs for RTL Generation: Quality, Reliability, and Failure Modes

Kansas State University(堪萨斯州立大学);NYU Tandon School of Engineering(纽约大学坦登工程学院);NYU Abu Dhabi(阿布扎比纽约大学)

Weimin Fu, Zeng Wang, Minghao Shao, Ramesh Karri, Muhammad Shafique, Johann Knechtel, Ozgur Sinanoglu, Xiaolong Guo

AI总结:

本文评估32个语言模型在202个Verilog任务上的RTL生成能力,通过硬件质量指数衡量合成与实现质量,发现不同模型在HQI指标上的表现差异显著,揭示了专有模型与开源模型在合成故障上的系统性差异。

摘要:

RTL生成不仅仅是代码综合。设计必须语法正确、可综合、正确且硬件高效。最先进的评估仅停留在功能正确性,未测量综合和实现质量。本文评估32个语言模型在202个Verilog任务上的表现,使用结合后综合面积、延迟和专家参考相关警告的硬件质量指数(HQI),在Nangate45 45nm流程中进行。三种性能区域显现:14个前沿模型达到HQI>66,Gemini-3-Pro以87.5%覆盖率和85.1 HQI领先;15个模型集中在43-66 HQI;3个低于43。最佳五能力与单次尝试质量之间的差距跨度3.7-22.1 HQI点,限制了集成到代理流程中。195种综合故障的分类揭示了系统性分歧:专有模型通过细化错误和综合超时在后期失败;开源模型因缺少模块包装器和不可综合构造而早期失败,这种模式与训练语料库偏向模拟而非综合级RTL一致。

英文摘要:

RTL generation is more than code synthesis. Designs must be syntactically valid, synthesizable, correct, hardware-efficient. SOTA evaluations stop at functional correctness and do not measure synthesis and implementation quality. This paper evaluates 32 language models on 202 Verilog tasks from VerilogEval and RTLLM using the Hardware Quality Index (HQI) that combines post-synthesis area, delay, and warnings related to expert references in a Nangate45 45\,nm flow. Three performance regimes emerge: 14 frontier models achieve HQI $>$ 66, led by Gemini-3-Pro at 87.5\% coverage and 85.1 HQI; 15 models cluster 43--66 HQI; 3 are below 43. Gap between best-of-five capability and single-attempt quality spans 3.7--22.1 HQI points, limiting integration into agentic pipelines. A taxonomy of 195 synthesis failures reveals systematic divergence: proprietary models fail late through elaboration errors and synthesis timeout; open models fail early often due to missing module wrappers and non-synthesizable constructs, a pattern consistent with training corpora skewed toward simulation over synthesis-grade RTL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑