arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2411.03471cs.ARcs.CL
收藏

MetRex:使用LLM进行Verilog代码指标推理的基准

MetRex: A Benchmark for Verilog Code Metric Reasoning Using LLMs

Brown University(布朗大学);School of Engineering(工程学院)

Manar Abdelatty, Jingxiao Ma, Sherief Reda

AI总结:

本文提出MetRex基准,包含25,868个Verilog设计及综合后指标,结合思维链模板评估并提升了LLM对Verilog代码综合后指标的推理能力,优于现有回归模型。

摘要:

大型语言模型(LLMs)已被应用于各种硬件设计任务,包括Verilog代码生成、EDA工具脚本编写和RTL错误修复。尽管进行了广泛的探索,但LLMs尚未用于HDL设计的综合后指标推理和估计任务。在本文中,我们评估了LLMs对Verilog设计的综合后指标进行推理的能力。我们引入了MetRex,这是一个大规模数据集,包含25,868个Verilog HDL设计及其相应的综合后指标,即面积、延迟和静态功耗。MetRex结合了思维链(CoT)模板,以增强LLMs对这些指标的推理能力。大量实验表明,监督微调(SFT)使LLM的推理能力在面积、延迟和静态功耗上分别平均提升了37.0%、25.3%和25.7%。虽然SFT提高了在我们基准上的性能,但它仍远未达到最佳结果,尤其是在复杂问题上。与最先进的回归模型相比,我们的方法为17.4%的更多设计提供了准确的综合后预测(在5%的误差范围内),此外通过消除预处理的需求提供了1.7倍的加速。这项工作为推进基于LLM的Verilog代码指标推理奠定了基础。

英文摘要:

Large Language Models (LLMs) have been applied to various hardware design tasks, including Verilog code generation, EDA tool scripting, and RTL bug fixing. Despite this extensive exploration, LLMs are yet to be used for the task of post-synthesis metric reasoning and estimation of HDL designs. In this paper, we assess the ability of LLMs to reason about post-synthesis metrics of Verilog designs. We introduce MetRex, a large-scale dataset comprising 25,868 Verilog HDL designs and their corresponding post-synthesis metrics, namely area, delay, and static power. MetRex incorporates a Chain of Thought (CoT) template to enhance LLMs' reasoning about these metrics. Extensive experiments show that Supervised Fine-Tuning (SFT) boosts the LLM's reasoning capabilities on average by 37.0\%, 25.3\%, and 25.7\% on the area, delay, and static power, respectively. While SFT improves performance on our benchmark, it remains far from achieving optimal results, especially on complex problems. Comparing to state-of-the-art regression models, our approach delivers accurate post-synthesis predictions for 17.4\% more designs (within a 5\% error margin), in addition to offering a 1.7x speedup by eliminating the need for pre-processing. This work lays the groundwork for advancing LLM-based Verilog code metric reasoning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑