arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.01475cs.LGcond-mat.mtrl-scics.AIphysics.chem-ph
收藏

大语言模型是超人类化学家吗?

Are large language models superhuman chemists?

Friedrich Schiller University Jena(弗里德里希·席勒大学耶拿分校);Helmholtz Institute for Polymers in Energy Applications Jena (HIPOLE Jena)(耶拿亥姆霍兹能源应用聚合物研究所);QpiVolta Technologies Pvt Ltd(QpiVolta技术私人有限公司);Laboratory of Artificial Chemical Intelligence (LIAC)(人工化学智能实验室);Ecole Polytechnique Fédérale de Lausanne (EPFL)(洛桑联邦理工学院);National Centre of Competence in Research (NCCR) Catalysis(国家催化研究能力中心)
Adrian Mirza, Nawaf Alampara, Sreekanth Kunchapu, Martiño Ríos-García, Benedict Emoekabu, Aswanth Krishnan, Tanya Gupta, Mara Schilling-Wilhelmi, Macjonathan Ok… 展开作者

Adrian Mirza, Nawaf Alampara, Sreekanth Kunchapu, Martiño Ríos-García, Benedict Emoekabu, Aswanth Krishnan, Tanya Gupta, Mara Schilling-Wilhelmi, Macjonathan Okereke, Anagha Aneesh, Amir Mohammad Elahi, Mehrdad Asgari, Juliane Eberhardt, Hani M. Elbeheiry, María Victoria Gil, Maximilian Greiner, Caroline T. Holick, Christina Glaubitz, Tim Hoffmann, Abdelrahman Ibrahim, Lea C. Klepsch, Yannik Köster, Fabian Alexander Kreth, Jakob Meyer, Santiago Miret, Jan Matthias Peschel, Michael Ringleb, Nicole Roesner, Johanna Schreiber, Ulrich S. Schubert, Leanne M. Stafast, Dinga Wonanke, Michael Pieler, Philippe Schwaller, Kevin Maik Jablonka

AI总结:

本研究提出自动化评估框架ChemBench,通过超2700组问答对测试大语言模型的化学能力,发现顶尖模型平均表现优于人类化学家,但存在基础任务薄弱、预测过度自信的问题,为模型优化与化学教育调整提供参考。

摘要:

大语言模型(LLM)因具备处理人类语言、执行未经过显式训练的任务的能力,引发了广泛关注。然而,我们对大语言模型的化学能力仅具备有限的系统性认知,而这种认知是改进模型、规避潜在危害的必要前提。本文提出“ChemBench”——一个自动化评估框架,用于对比前沿大语言模型与化学家的专业水平,评估大语言模型的化学知识与推理能力。我们整理了超过2700组问答对,对主流的开源与闭源大语言模型进行了评估,结果显示表现最优的模型平均水平超过了本研究中表现最好的人类化学家。不过,这些模型在部分基础任务上表现不佳,且会给出过于自信的预测。这些发现既展现了大语言模型令人瞩目的化学能力,也凸显了需进一步研究以提升其安全性与实用性的必要性。研究结果还为化学教育的调整提供了参考,同时证明了基准测试框架在特定领域大语言模型评估中的价值。

英文摘要:

Large language models (LLMs) have gained widespread interest due to their ability to process human language and perform tasks on which they have not been explicitly trained. However, we possess only a limited systematic understanding of the chemical capabilities of LLMs, which would be required to improve models and mitigate potential harm. Here, we introduce "ChemBench," an automated framework for evaluating the chemical knowledge and reasoning abilities of state-of-the-art LLMs against the expertise of chemists. We curated more than 2,700 question-answer pairs, evaluated leading open- and closed-source LLMs, and found that the best models outperformed the best human chemists in our study on average. However, the models struggle with some basic tasks and provide overconfident predictions. These findings reveal LLMs' impressive chemical capabilities while emphasizing the need for further research to improve their safety and usefulness. They also suggest adapting chemistry education and show the value of benchmarking frameworks for evaluating LLMs in specific domains.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑