Pluto:评估LLM生成硬件代码效率的基准
Pluto: A Benchmark for Evaluating Efficiency of LLM-generated Hardware Code
AI总结:
针对现有基准忽视硬件设计效率的问题,提出Pluto基准与评估框架,含114个问题及帕累托最优参考,实验显示LLM功能正确性高但综合效率落后,需效率感知评估推动研究。
摘要:
大型语言模型(LLMs)正越来越多地被用于自动化硬件设计任务,包括生成Verilog代码。虽然早期基准主要关注功能正确性,但高效的硬件设计还要求对面积、延迟和功耗等综合指标进行额外优化。现有基准在全面评估这些方面存在不足:它们往往缺乏优化的基线或用于验证的测试平台。为解决这些差距,我们提出了Pluto,一个旨在评估LLM生成的Verilog设计效率的基准和评估框架。Pluto提供了包含114个问题的全面评估集,配有自检测试平台和多个帕累托最优参考实现。实验结果表明,最先进的LLMs能够实现较高的功能正确性,在pass@1上达到78.3%,但其综合效率仍落后于专家编写的实现,在eff@1上面积效率为63.8%,延迟效率为65.9%,功耗效率为64.0%。这凸显了诸如Pluto之类的效率感知评估框架对于推动硬件聚焦的LLM研究进步的必要性。
英文摘要:
Large Language Models (LLMs) are increasingly used to automate hardware design tasks, including the generation of Verilog code. While early benchmarks focus primarily on functional correctness, efficient hardware design demands additional optimization for synthesis metrics such as area, delay, and power. Existing benchmarks fall short in evaluating these aspects comprehensively: they often lack optimized baselines or testbenches for verification. To address these gaps, we present Pluto, a benchmark and evaluation framework designed to assess the efficiency of LLM-generated Verilog designs. Pluto presents a comprehensive evaluation set of 114 problems with self-checking testbenches and multiple Pareto-optimal reference implementations. Experimental results show that state-of-the-art LLMs can achieve high functional correctness, reaching 78.3\% at pass@1, but their synthesis efficiency still lags behind expert-crafted implementations, with area efficiency of 63.8\%, delay efficiency of 65.9\%, and power efficiency of 64.0\% at eff@1. This highlights the need for efficiency-aware evaluation frameworks such as Pluto to drive progress in hardware-focused LLM research.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。