arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.03708cs.PLcs.AI
收藏

MHRC-Bench: 一个多语言硬件仓库级代码补全基准

MHRC-Bench: A Multilingual Hardware Repository-Level Code Completion benchmark

School of Computing, National University of Singapore(新加坡国立大学计算机学院)

Qingyun Zou, Jiahao Cui, Nuo Chen, Bingsheng He, Weng-Fai Wong

AI总结:

MHRC-Bench是首个多语言硬件仓库级代码补全基准,涵盖三种硬件设计编码风格,通过代码结构和硬件导向语义标签进行补全任务评估。

摘要:

大型语言模型(LLMs)在通用编程语言的代码补全任务中取得了强劲表现。然而,现有仓库级代码补全基准几乎只关注软件代码,且大多忽视了硬件描述语言。在本工作中,我们提出了MHRC-Bench,包括MHRC-Bench-Train和MHRC-Bench-Eval,这是首个针对多语言硬件代码补全的仓库级基准。我们的基准针对补全任务,涵盖三种主要的硬件设计编码风格。每个补全目标都带有基于具体语法树分析的代码结构级和硬件导向的语义标签。我们在MHRC-Bench-Eval上对模型进行了全面评估。全面的评估结果和分析证明了MHRC-Bench的有效性。

英文摘要:

Large language models (LLMs) have achieved strong performance on code completion in general-purpose programming languages, but existing repository-level benchmarks focus almost exclusively on software code and largely overlook hardware description languages (HDLs). We present \textbf{MHRC-Bench}, consisting of \textbf{MHRC-Bench-Train} and \textbf{MHRC-Bench-Eval}, the first benchmark for multilingual hardware code completion at the repository level with functional evaluation. MHRC-Bench covers three major hardware design coding styles and annotates each completion target with code-structure-level and hardware-oriented semantic labels derived from concrete syntax tree (CST) analysis. Post-training billion-scale code LLMs on MHRC-Bench-Train enables smaller open-weight models to outperform much larger general-purpose models under matched context windows. Beyond aggregate accuracy, our analysis surfaces four findings that distinguish hardware from software code completion: \emph{(i)}~performance varies sharply across HDLs, with high-level synthesis (HLS) easiest and VHDL hardest in most settings across functional pass rate, exact match (EM), and edit similarity (ES); \emph{(ii)}~hardware accuracy is non-monotonic in code-structure depth, in contrast with the monotonic depth--accuracy curve known for software; \emph{(iii)}~accuracy plateaus once the input context exceeds 2{,}048 tokens, while software completion continues to benefit from longer context; and \emph{(iv)}~certain repository-level retrievers tuned on software underperform a no-retrieval baseline on HDL repositories because their structural and dependency assumptions are violated by hardware code.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑