arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.19006cs.AIquant-ph
收藏

评估大型语言模型在量子力学中的表现：跨多样模型和任务的比较研究

Evaluating Large Language Models on Quantum Mechanics: A Comparative Study Across Diverse Models and Tasks

Quantum Science and Technology Laboratory, Physical Research Laboratory, Navrangpura, Ahmedabad 380009, India（量子科学与技术实验室，物理研究实验室，Ahmedabad）；Indian Institute of Technology Gandhinagar, Palaj, Gandhinagar 382055, India（印度理工学院冈达塞瓦尔）

S. K. Rithvik

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究评估了不同大型语言模型在量子力学问题上的表现，揭示了模型层级、任务难度及工具增强效果的差异，提供了可重复的基准和性能分析。

摘要：

我们系统地评估了大型语言模型在量子力学问题解决中的表现。我们的研究评估了来自五个供应商（OpenAI、Anthropic、Google、Alibaba、DeepSeek）的15个模型，涵盖三个能力层级，在20个任务上进行评估，涵盖推导、创造问题、非标准概念和数值计算，包括900个基准任务和75个工具增强评估任务。结果揭示了清晰的层级分层：旗舰模型的平均准确率为81%，比中层模型（77%）和快速模型（67%）分别高出4个百分点和14个百分点。任务难度模式呈现出明显差异：推导任务表现最高（平均92%，旗舰模型为100%），而数值计算仍然最具有挑战性（42%）。工具增强在数值任务上的效果具有任务依赖性：总体改善（+4.4个百分点）在3倍token成本下，掩盖了范围从+29个百分点的收益到-16个百分点的退化。在三个运行的可重复性分析中，量化了6.3个百分点的平均方差，旗舰模型表现出极高的稳定性（GPT-5达到零方差），而专门化的模型需要多轮评估。本工作贡献：（i）具有自动验证的量子力学基准；（ii）系统评估，量化基于层级的性能等级；（iii）经验分析工具增强的权衡；（iv）可重复性特征。所有任务、验证器和结果均已公开发布。

英文摘要：

We present a systematic evaluation of large language models on quantum mechanics problem-solving. Our study evaluates 15 models from five providers (OpenAI, Anthropic, Google, Alibaba, DeepSeek) spanning three capability tiers on 20 tasks covering derivations, creative problems, non-standard concepts, and numerical computation, comprising 900 baseline and 75 tool-augmented assessments. Results reveal clear tier stratification: flagship models achieve 81\% average accuracy, outperforming mid-tier (77\%) and fast models (67\%) by 4pp and 14pp respectively. Task difficulty patterns emerge distinctly: derivations show highest performance (92\% average, 100\% for flagship models), while numerical computation remains most challenging (42\%). Tool augmentation on numerical tasks yields task-dependent effects: modest overall improvement (+4.4pp) at 3x token cost masks dramatic heterogeneity ranging from +29pp gains to -16pp degradation. Reproducibility analysis across three runs quantifies 6.3pp average variance, with flagship models demonstrating exceptional stability (GPT-5 achieves zero variance) while specialized models require multi-run evaluation. This work contributes: (i) a benchmark for quantum mechanics with automatic verification, (ii) systematic evaluation quantifying tier-based performance hierarchies, (iii) empirical analysis of tool augmentation trade-offs, and (iv) reproducibility characterization. All tasks, verifiers, and results are publicly released.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑