Large Language Models (LLMs) create exciting possibilities for powerful language processing tools to accelerate research in materials science. While LLMs have great potential to accelerate materials understanding and discovery, they currently fall short in being practical materials science tools. In this position paper, we show relevant failure cases of LLMs in materials science that reveal current limitations of LLMs related to comprehending and reasoning over complex, interconnected materials science knowledge. Given those shortcomings, we outline a framework for developing Materials Science LLMs (MatSci-LLMs) that are grounded in materials science knowledge and hypothesis generation followed by hypothesis testing. The path to attaining performant MatSci-LLMs rests in large part on building high-quality, multi-modal datasets sourced from scientific literature where various information extraction challenges persist. As such, we describe key materials science information extraction challenges which need to be overcome in order to build large-scale, multi-modal datasets that capture valuable materials science knowledge. Finally, we outline a roadmap for applying future MatSci-LLMs for real-world materials discovery via: 1. Automated Knowledge Base Generation; 2. Automated In-Silico Material Design; and 3. MatSci-LLM Integrated Self-Driving Materials Laboratories.
Are LLMs Ready to Assist Physicians? PhysAssistBench for Interactive Doctor-Patient-EHR Assistance
Aalto University(阿尔托大学);Tencent(腾讯);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Hong Kong Polytechnic University(香港理工大学);Aarhus University(奥胡斯大学);Technical University of Munich(慕尼黑工业大学)
LooGLE v2: Are LLMs Ready for Real World Long Dependency Challenges?
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);National Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室,BIGAI);School of Computer Science and Technology, Beijing Institute of Technology(北京理工大学计算机科学与技术学院)
Can LLMs Clean Up Your Mess? A Survey of Application-Ready Data Preparation with LLMs
Shanghai Jiao Tong University(上海交通大学);Tsinghua University(清华大学);Microsoft Research(微软研究院);MIT CSAIL(麻省理工学院计算机科学与人工智能实验室);Shanghai AI Laboratory(上海人工智能实验室);Xiaohongshu Inc.(小红书公司);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Alibaba Group(阿里巴巴集团)
Ready to Translate, Not to Represent? Bias and Performance Gaps in Multilingual LLMs Across Language Families and Domains
United International University(国际联合大学);Qatar Computing Research Institute(卡塔尔计算研究所);Amazon GenAI(亚马逊生成人工智能);University of Virginia(弗吉尼亚大学);BRAC University(布拉克大学)
InSaAF: Incorporating Safety through Accuracy and Fairness | Are LLMs ready for the Indian Legal Domain?
Centre for Responsible AI, Indian Institute of Technology Madras(印度马德拉斯理工学院负责任人工智能中心);International Institute of Information Technology, Hyderabad(海德拉巴国际信息技术学院);AmexAI Labs, American Express(美国运通AmexAI实验室);Wadhwani School of Data Science and AI, Indian Institute of Technology Madras(印度马德拉斯理工学院瓦德瓦尼数据科学与人工智能学院)
ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition
Fudan University(复旦大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Nanyang Technological University(南洋理工大学);University of New South Wales(新南威尔士大学);National University of Singapore(新加坡国立大学);Wuhan University(武汉大学)
Attribution-Guided Pruning for Insight and Control: Circuit Discovery and Targeted Correction in Small-scale LLMs
Data Science Center ScaDS.AI, Universität Leipzig(ScaDS.AI数据科学中心,莱比锡大学);Department of Artificial Intelligence, Fraunhofer Heinrich-Hertz-Institute(人工智能系,弗劳恩霍夫 Heinrich-Hertz 研究所);Department of Electrical Engineering and Computer Science, Technische Universität Berlin(电气工程与计算机科学系,柏林技术大学);BIFOLD - Berlin Institute for the Foundations of Learning and Data(柏林学习与数据基础研究所);Centre of eXplainable Artificial Intelligence, Technological University Dublin(可解释人工智能中心,都柏林技术大学)
Role of Large Language Models and Retrieval-Augmented Generation for Accelerating Crystalline Material Discovery: A Systematic Review
Bredesen Center for Interdisciplinary Research, University of Tennessee, Knoxville, USA(跨学科研究中心,田纳西大学,诺克斯维尔,美国);University of Tennessee-Oak Ridge Innovation Institute, University of Tennessee, Knoxville, USA(田纳西大学橡树岭创新研究所,田纳西大学,诺克斯维尔,美国)
An LLM agent for end-to-end computational materials discovery
Northwestern Polytechnical University(西北工业大学);National University of Singapore(新加坡国立大学);University of Toronto(多伦多大学);Chinese Academy of Sciences(中国科学院);Huazhong University of Science and Technology(华中科技大学);The University of Tokyo(东京大学)
Deploy-Master: Automating the Deployment of 50,000+ Agent-Ready Scientific Tools in One Day
DP Technology Beijing China(DP技术北京中国);AI for Science Institute Beijing China(AI for Science研究院北京中国);Shanghai Jiao Tong University Shanghai China(上海交通大学上海中国)
A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery
University of Washington(华盛顿大学);Texas A&M University(德克萨斯农工大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of California, Los Angeles(加利福尼亚大学洛杉矶分校)