Friedrich Schiller University Jena(弗里德里希·席勒大学耶拿分校);Helmholtz Institute for Polymers in Energy Applications Jena (HIPOLE Jena)(耶拿亥姆霍兹能源应用聚合物研究所);QpiVolta Technologies Pvt Ltd(QpiVolta技术私人有限公司);Laboratory of Artificial Chemical Intelligence (LIAC)(人工化学智能实验室);Ecole Polytechnique Fédérale de Lausanne (EPFL)(洛桑联邦理工学院);National Centre of Competence in Research (NCCR) Catalysis(国家催化研究能力中心)
Adrian Mirza, Nawaf Alampara, Sreekanth Kunchapu, Martiño Ríos-García, Benedict Emoekabu, Aswanth Krishnan, Tanya Gupta, Mara Schilling-Wilhelmi, Macjonathan Ok… 展开作者
Adrian Mirza, Nawaf Alampara, Sreekanth Kunchapu, Martiño Ríos-García, Benedict Emoekabu, Aswanth Krishnan, Tanya Gupta, Mara Schilling-Wilhelmi, Macjonathan Okereke, Anagha Aneesh, Amir Mohammad Elahi, Mehrdad Asgari, Juliane Eberhardt, Hani M. Elbeheiry, María Victoria Gil, Maximilian Greiner, Caroline T. Holick, Christina Glaubitz, Tim Hoffmann, Abdelrahman Ibrahim, Lea C. Klepsch, Yannik Köster, Fabian Alexander Kreth, Jakob Meyer, Santiago Miret, Jan Matthias Peschel, Michael Ringleb, Nicole Roesner, Johanna Schreiber, Ulrich S. Schubert, Leanne M. Stafast, Dinga Wonanke, Michael Pieler, Philippe Schwaller, Kevin Maik Jablonka
Large language models (LLMs) have gained widespread interest due to their ability to process human language and perform tasks on which they have not been explicitly trained.
However, we possess only a limited systematic understanding of the chemical capabilities of LLMs, which would be required to improve models and mitigate potential harm. Here, we introduce "ChemBench," an automated framework for evaluating the chemical knowledge and reasoning abilities of state-of-the-art LLMs against the expertise of chemists.
We curated more than 2,700 question-answer pairs, evaluated leading open- and closed-source LLMs, and found that the best models outperformed the best human chemists in our study on average. However, the models struggle with some basic tasks and provide overconfident predictions.
These findings reveal LLMs' impressive chemical capabilities while emphasizing the need for further research to improve their safety and usefulness. They also suggest adapting chemistry education and show the value of benchmarking frameworks for evaluating LLMs in specific domains.
Superhuman performance of a large language model on the reasoning tasks of a physician
Beth Israel Deaconess Medical Center(贝斯以色列女执事医疗中心);Harvard Medical School(哈佛医学院);Stanford University(斯坦福大学);Cambridge Health Alliance(剑桥健康联盟);Brigham and Women's Hospital(布列根和妇女医院);University of Minnesota Medical School(明尼苏达大学医学院);University of Maryland School of Medicine(马里兰大学医学院);Veterans Affairs Maryland Healthcare System(马里兰退伍军人医疗保健系统);VA Palo Alto Health Care System(帕洛阿尔托退伍军人医疗保健系统);Massachusetts Institute of Technology(麻省理工学院);University of Alberta(阿尔伯塔大学);Beth Israel Lahey Health(贝斯以色列拉海健康);Massachusetts General Hospital(麻省总医院);Microsoft Corp(微软公司)
A Review of Large Language Models and Autonomous Agents in Chemistry
FutureHouse Inc.(FutureHouse公司);University of Rochester(罗切斯特大学);School of Chemistry and Materials Science(化学与材料科学学院);Rochester Institute of Technology(罗切斯特理工学院)
From Words to Molecules: A Survey of Large Language Models in Chemistry
School of Computer Science and Engineering, Nanyang Technological University(南洋理工大学计算机科学与工程学院);City University of Hong Kong(香港城市大学);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院)
ChemToolAgent: The Impact of Tools on Language Agents for Chemistry Problem Solving
Department of Computer Science and Engineering, OSU(计算机科学与工程系,俄勒冈州立大学);Department of Biomedical Informatics, OSU(生物医学信息学系,俄勒冈州立大学);College of Pharmacy, OSU(药学院,俄勒冈州立大学)
A GPT-4 Reticular Chemist for Guiding MOF Discovery
University of California, Berkeley(加州大学伯克利分校);Kavli Energy Nanoscience Institute(卡夫利能源纳米科学研究所);Bakar Institute of Digital Materials for the Planet(巴卡尔全球数字材料研究所);College of Computing, Data Science, and Society(计算、数据科学与社会学院);Department of Chemistry(化学系);Department of Electrical Engineering and Computer Sciences(电气工程与计算机科学系);Department of Statistics(统计学系);Department of Mathematics(数学系);School of Information(信息学院);King Abdulaziz City for Science and Technology(阿卜杜勒阿齐兹国王科学技术城)
Structured Chemistry Reasoning with Large Language Models
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Shanghai Jiao Tong University(上海交通大学);New York University(纽约大学);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of California San-Diego(加利福尼亚大学圣迭戈分校);University of Washington(华盛顿大学)
MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypotheses
Nanyang Technological University(南洋理工大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学);Wuhan University(武汉大学);University of New South Wales(新南威尔士大学);GreenDynamics;Singapore University of Technology and Design(新加坡科技设计大学)
Chemist-X: Large Language Model-empowered Agent for Reaction Condition Recommendation in Chemical Synthesis
The Chinese University of Hong Kong(香港中文大学);ZJU-Hangzhou Global Scientific and Technological Innovation Center(浙大杭州国际科创中心);Zhejiang University(浙江大学);Zhejiang Lab(之江实验室)
International Digital Economy Academy (IDEA)(国际数字经济学院(IDEA));Hong Kong University of Science and Technology(香港科技大学);Yale University(耶鲁大学);National University of Singapore(新加坡国立大学)
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Fudan University(复旦大学);Shanghai Jiao Tong University(上海交通大学);Wuhan University(武汉大学);The Hong Kong Polytechnic University(香港理工大学);The Chinese University of Hong Kong(香港中文大学)
ChemVTS-Bench: Evaluating Visual-Textual-Symbolic Reasoning of Multimodal Large Language Models in Chemistry
Renmin University of China(中国人民大学);Beijing University of Posts and Telecommunications(北京邮电大学);South China University of Technology(华南理工大学);Gaotu Techedu Inc(高图科技公司)
Bridging the Gap between Expert and Language Models: Concept-guided Chess Commentary Generation and Evaluation
Graduate School of AI, POSTECH(浦项科技大学人工智能研究生院);Department of Computer Science and Engineering, POSTECH(浦项科技大学计算机科学与工程系);Department of Chemical Engineering, POSTECH(浦项科技大学化学工程系);KRAFTON
KALE-LM-Chem: Vision and Practice Toward an AI Brain for Chemistry
University of Science and Technology of China(科学技术大学);ShanghaiTech University(上海科技大学);USTC Knowledge Computing Lab(USTC知识计算实验室);State Key Laboratory of Communication Content Cognition People's Daily Online(通信内容认知国家重点实验室)
ChemEval: A Comprehensive Multi-Level Chemical Evaluation for Large Language Models
University of Science and Technology of China(中国科学技术大学);State Key Laboratory of Cognitive Intelligence(认知智能国家重点实验室);Artificial Intelligence Research Institute, iFLYTEK Co., Ltd.(讯飞股份有限公司人工智能研究院);iFLYTEK AI Research (Hebei)(讯飞人工智能研究院(河北))
Safety challenges of AI in medicine in the era of large language models
National University of Singapore(新加坡国立大学);University of Cambridge(剑桥大学);Mila-Quebec AI Institute(米拉-魁北克人工智能研究所);Harvard Medical School(哈佛医学院);Massachusetts General Hospital(麻省总医院);Yale University(耶鲁大学);Brigham and Women’s Hospital/Dana-Farber Cancer Institute(布莱根妇女医院/达纳-法伯癌症研究所);The Hong Kong University of Science and Technology(香港科技大学);Stanford University(斯坦福大学)
Generative Evaluation of Complex Reasoning in Large Language Models
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Institute for AI Industry Research, Tsinghua University(清华大学人工智能产业研究院);Computer Science Department, Stanford University(斯坦福大学计算机科学系)
MolProphecy: Bridging Medicinal Chemists' Knowledge and Molecular Pre-Trained Models via a Multi-Modal Framework
College of Computer Science and Technology, Changchun University of Science and Technology(长春理工大学计算机科学与技术学院);Department of Chemical and Biological Engineering, The Hong Kong University of Science and Technology(香港科技大学化学与生物工程系);Key Laboratory of Symbolic Computation and Knowledge Engineering, Ministry of Education, Jilin University(吉林大学符号计算与知识工程重点实验室);Communication University of China(通信大学);Beijing Life Science Academy(北京生命科学研究院);College of Computer Science and Technology, Jilin University(吉林大学计算机科学与技术学院)
Image and Data Mining in Reticular Chemistry Using GPT-4V
University of California, Berkeley(加州大学伯克利分校);Kavli Energy Nanoscience Institute(卡夫利能源纳米科学研究所);Bakar Institute of Digital Materials for the Planet(巴卡尔行星数字材料研究所);Stanford University(斯坦福大学);King Abdulaziz City for Science and Technology(阿卜杜勒阿齐兹国王科技城)