大语言模型的物理模拟能力
Physics simulation capabilities of LLMs
AI总结：
本文评估了GPT4等最先进大语言模型在博士级计算物理问题上的表现，发现其零样本通过率有限，并识别了主要失效模式与改进方向。
摘要：
大语言模型（LLMs）能够解决部分本科至研究生水平的物理教科书问题，并且擅长编程。将这两种能力结合起来，有朝一日可能使人工智能系统能够模拟和预测物理世界。我们对最先进（SOTA）的大语言模型在博士级至研究级计算物理问题上的表现进行了评估。我们以使用文档完善且广泛使用的软件包为条件来引导大语言模型的生成，以激发其在物理学和天体物理学领域的编程能力。我们贡献了约50个原创且具有挑战性的问题，涵盖天体力学（使用REBOUND）、恒星物理（使用MESA）、一维流体动力学（使用Dedalus）和非线性动力学（使用SciPy）。由于我们的问题不存在唯一解，我们通过若干软性指标来评估大语言模型的表现：包含不同类型错误（编码、物理、必要性和充分性）的代码行计数，以及一个更具“教育意义”的通过/不通过指标，该指标侧重于捕捉所给问题中关键的物理要素。正如预期的那样，当今最先进的大语言模型（GPT4）在零样本设置下未能通过我们的大多数问题，尽管约40%的解答有可能获得及格分数。所生成的代码行中约70-90%是必要、充分且正确的（编码与物理方面）。物理错误和编码错误最为常见，同时存在一些不必要或不充分的代码行。我们观察到不同问题类别和难度之间存在显著差异。我们识别出GPT4在计算物理领域中的若干失效模式。我们的侦察性工作提供了当前经典物理计算能力的快照，并指出了明显的改进目标，如果人工智能系统要在物理模拟能力方面达到基本的自主水平。
英文摘要：
[Abridged abstract] Large Language Models (LLMs) can solve some undergraduate-level to graduate-level physics textbook problems and are proficient at coding. Combining these two capabilities could one day enable AI systems to simulate and predict the physical world. We present an evaluation of state-of-the-art (SOTA) LLMs on PhD-level to research-level computational physics problems. We condition LLM generation on the use of well-documented and widely-used packages to elicit coding capabilities in the physics and astrophysics domains. We contribute $\sim 50$ original and challenging problems in celestial mechanics (with REBOUND), stellar physics (with MESA), 1D fluid dynamics (with Dedalus) and non-linear dynamics (with SciPy). Since our problems do not admit unique solutions, we evaluate LLM performance on several soft metrics: counts of lines that contain different types of errors (coding, physics, necessity and sufficiency) as well as a more "educational" Pass-Fail metric focused on capturing the salient physical ingredients of the problem at hand. As expected, today's SOTA LLM (GPT4) zero-shot fails most of our problems, although about 40\% of the solutions could plausibly get a passing grade. About $70-90 \%$ of the code lines produced are necessary, sufficient and correct (coding \& physics). Physics and coding errors are the most common, with some unnecessary or insufficient lines. We observe significant variations across problem class and difficulty. We identify several failure modes of GPT4 in the computational physics domain. Our reconnaissance work provides a snapshot of current computational capabilities in classical physics and points to obvious improvement targets if AI systems are ever to reach a basic level of autonomy in physics simulation capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
