CreativEval：评估基于大语言模型的硬件代码生成的创造性
CreativEval: Evaluating Creativity of LLM-Based Hardware Code Generation
AI总结：
针对现有LLM硬件代码生成评估仅关注功能正确性、缺乏创造性量化方法的问题，提出CreativeEval框架，从四个子维度评估LLM硬件设计生成的创造性，实验发现GPT-3.5在该任务中创造性最强。
摘要：
大语言模型（LLM）已被证明在代码生成方面高效且有效，因此被应用于硬件设计流程中。此前评估LLM寄存器传输级代码生成能力的研究仅关注功能正确性。然而，这些LLM的创造性，即生成新颖独特解决方案的能力，是一个尚未被充分理解的指标，部分原因在于该质量难以量化。为填补这一研究空白，我们提出了CreativeEval，一个用于评估LLM在硬件设计生成场景下创造性的框架。我们通过多种提示工程和后处理技术，量化了四个创造性子维度：流畅性、灵活性、独创性和精细性。随后我们在该创造性指标上评估了多个主流LLM（包括GPT模型、CodeLlama和VeriGen），结果表明GPT-3.5是生成硬件设计时最具创造性的模型。
英文摘要：
Large Language Models (LLMs) have proved effective and efficient in generating code, leading to their utilization within the hardware design process. Prior works evaluating LLMs' abilities for register transfer level code generation solely focus on functional correctness. However, the creativity associated with these LLMs, or the ability to generate novel and unique solutions, is a metric not as well understood, in part due to the challenge of quantifying this quality. To address this research gap, we present CreativeEval, a framework for evaluating the creativity of LLMs within the context of generating hardware designs. We quantify four creative sub-components, fluency, flexibility, originality, and elaboration, through various prompting and post-processing techniques. We then evaluate multiple popular LLMs (including GPT models, CodeLlama, and VeriGen) upon this creativity metric, with results indicating GPT-3.5 as the most creative model in generating hardware designs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
