arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.10092cs.CL
收藏

量子审计:评估大语言模型在量子计算上的推理极限

Quantum-Audit: Evaluating the Reasoning Limits of LLMs on Quantum Computing

Fordham University(福特汉姆大学);University of Michigan-Dearborn(密歇根大学-迪尔本分校);Stevens Institute of Technology(史蒂文斯理工学院)

Mohamed Afane, Kayla Laufer, Wenqi Wei, Ying Mao, Junaid Farooq, Ying Wang, Juntao Chen

AI总结:

Quantum-Audit通过2700个问题评估大语言模型在量子计算概念理解上的推理能力,发现顶级模型在专家问题上表现不如LLM生成问题,且在高级主题上准确率显著下降。

摘要:

语言模型已成为量子计算教育和研究中的实用工具,从总结技术论文到解释理论概念,再到回答领域内最新发展的问答。尽管现有的基准测试评估了量子代码生成和电路设计,但对量子计算概念的理解尚未系统地进行测量。Quantum-Audit通过涵盖核心量子计算主题的2700个问题填补了这一空白。我们评估了26家领先组织的模型。我们的基准测试包括1000个专家编写的问题、1000个由LLM提取自研究论文并通过专家验证的问题,以及另外700个问题,其中包括350个开放式问题和350个包含错误前提的问题,以测试模型是否能纠正错误的假设。人类参与者在23%至86%之间得分,专家平均得分为74%。表现最好的模型超过了专家平均分,Claude Opus 4.5达到84%的准确率,但顶级模型在专家编写的问题上比LLM生成的问题平均准确率低12个百分点。在高级主题上,表现进一步下降,安全问题的准确率降至73%。此外,模型经常接受并强化问题中嵌入的错误前提,而不是识别它们,在这些关键推理任务上的准确率低于66%。

英文摘要:

Language models have become practical tools for quantum computing education and research, from summarizing technical papers to explaining theoretical concepts and answering questions about recent developments in the field. While existing benchmarks evaluate quantum code generation and circuit design, their understanding of quantum computing concepts has not been systematically measured. Quantum-Audit addresses this gap with 2,700 questions covering core quantum computing topics. We evaluate 26 models from leading organizations. Our benchmark comprises 1,000 expert-written questions, 1,000 questions extracted from research papers using LLMs and validated by experts, plus an additional 700 questions including 350 open-ended questions and 350 questions with false premises to test whether models can correct erroneous assumptions. Human participants scored between 23% and 86%, with experts averaging 74%. Top-performing models exceeded the expert average, with Claude Opus 4.5 reaching 84% accuracy, though top models showed an average 12-point accuracy drop on expert-written questions compared to LLM-generated ones. Performance declined further on advanced topics, dropping to 73% on security questions. Additionally, models frequently accepted and reinforced false premises embedded in questions instead of identifying them, with accuracy below 66% on these critical reasoning tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑