QHackBench：使用 PennyLane 黑客松挑战评测大语言模型的量子代码生成能力
QHackBench: Benchmarking Large Language Models for Quantum Code Generation Using PennyLane Hackathon Challenges
AI总结：
本文构建源自 QHack 的 QHackBench 量子代码生成基准，评测 LLM 在普通提示与 RAG 下的表现，并提出多智能体迭代修正流程。
摘要：
大语言模型（LLM）近期的进展已展现出在代码生成方面的强大潜力，但其在量子计算中的有效性仍未得到充分研究。本文使用来自量子黑客松（QHack）的真实挑战，对基于 PennyLane 的量子代码生成 LLM 进行基准评测。我们引入 QHackBench，这是一个源自 QHack 竞赛的新型基准数据集，并在普通提示和检索增强生成（RAG）条件下评估模型性能。我们的结构化评估框架在不同挑战难度上评估功能正确性、语法有效性和执行成功率。结果表明，借助增强版 PennyLane 数据集的 RAG 增强模型生成的结果与标准提示大致相似，尤其是在复杂量子算法方面。此外，我们提出一种多智能体评估流程，可迭代修正错误解决方案，进一步提高执行成功率。为促进进一步研究，我们承诺公开发布 QHackBench 及其评估框架和实验结果，从而推动 AI 辅助量子编程持续进步。
英文摘要：
Recent advances in Large Language Models (LLMs) have demonstrated strong potential in code generation, yet their effectiveness in quantum computing remains underexplored. This paper benchmarks LLMs for PennyLane-based quantum code generation using real-world challenges from the Quantum Hackathon (QHack). We introduce QHackBench, a novel benchmark dataset derived from QHack competitions, and evaluate model performance under vanilla prompting and Retrieval-Augmented Generation (RAG). Our structured evaluation framework assesses functional correctness, syntactic validity, and execution success across varying challenge difficulties. Results indicate that RAG-enhanced models, supplemented with an augmented PennyLane dataset, approximately generate similar results as the standard prompting, particularly in complex quantum algorithms. Additionally, we introduce a multi-agent evaluation pipeline that iteratively refines incorrect solutions, further enhancing execution success rates. To foster further research, we commit to publicly releasing QHackBench, along with our evaluation framework and experimental results, enabling continued advancements in AI-assisted quantum programming.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
