跨连续软件开发工具包版本对大语言模型生成的量子代码中的 API 漂移进行基准测试
Benchmarking API Drift in LLM-Generated Quantum Code Across Successive SDK Versions
AI总结:
研究大语言模型生成的量子代码能否可靠适配特定 SDK 版本(API 漂移问题),引入量子 API 漂移基准,以 Qiskit 为例评估 17 个模型,揭示版本对齐是量子代码生成评估新维度及 API 漂移恢复情况。
摘要:
大语言模型可生成看似合理的量子代码,但能否可靠适配用户所需特定软件开发工具包 (SDK) 版本尚不明晰。我们将此问题作为 API 漂移进行研究,并引入量子 - API - 漂移,这是一个用于衡量版本保真度的基准,在此定义为在请求的 SDK 版本上的执行成功、跨版本兼容性、失败模式以及大语言模型生成的量子 SDK 代码中的文档引导修复。我们使用 Qiskit(一个代表性的量子 SDK,在 v0.43、v1.3 和 v2.0 之间经历了重大接口更改)实例化该基准。我们在 50 个任务上对 17 个模型进行评估,每个提示有 3 个样本,每个模型产生 450 个生成样本和 1350 次执行。16 个模型在匹配的 REST API 设置中进行测试,输出上限为 1024 个令牌,而 GPT - 5.4(Codex CLI)作为参考配置单独报告。在 16 个匹配的 REST 模型中,对角 Pass@1 的范围从 0.02 到 0.85。Claude Opus 4.7 在 v0.43 和 v2.0 上最强,而 Grok 4.20 在 v1.3 上最强,为 0.513。错误概况因模型强度而系统地不同:较弱的模型主要因导入中断而失败,而较强的模型更常达到弃用级别的失败。文档引导修复在总体修复尝试中的成功率为 0.19 至 0.59,并且对于迁移到 v2.0 始终比对迁移到 v1.3 更有效。基准工件可在这个 https URL 上公开获取。这些结果表明,版本对齐是量子代码生成的一个独特评估轴,并且即使有迁移指导,API 漂移仍只能部分恢复。
英文摘要:
Large language models can generate plausible quantum code, but it is unclear whether they can reliably target the specific software development kit (SDK) version requested by the user. We study this problem as API drift and introduce quantum-api-drift, a benchmark for measuring version fidelity, defined here as execution success on the requested SDK version, cross-version compatibility, failure modes, and documentation-guided repair in LLM-generated quantum SDK code. We instantiate the benchmark with Qiskit, a representative quantum SDK that underwent substantial interface changes across v0.43, v1.3, and v2.0. We evaluate 17 models on 50 tasks with 3 samples per prompt, yielding 450 generated samples and 1,350 executions per model. Sixteen models are tested in a matched REST API setting with a 1024-token output cap, while GPT-5.4 (Codex CLI) is reported separately as a reference configuration. Across the 16 matched REST models, diagonal Pass@1 ranges from 0.02 to 0.85. Claude Opus 4.7 is strongest on v0.43 and v2.0, while Grok 4.20 is strongest on v1.3 at 0.513. Error profiles differ systematically by model strength: weaker models fail mainly with broken imports, while stronger models more often reach deprecation-level failures. Documentation-guided repair succeeds for 0.19 to 0.59 of repair attempts overall and is consistently much more effective for migration to v2.0 than to v1.3. The benchmark artifacts are publicly available at https://github.com/arasyi/quantum-api-drift. These results show that version alignment is a distinct evaluation axis for quantum code generation and that API drift remains only partly recoverable even with migration guidance.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。