Scientific reasoning, the process through which humans apply logic, evidence, and critical thinking to explore and interpret scientific phenomena, is essential in advancing knowledge reasoning across diverse fields. However, despite significant progress, current scientific reasoning models still struggle with generalization across domains and often fall short of multimodal perception. Multimodal Large Language Models (MLLMs), which integrate text, images, and other modalities, present an exciting opportunity to overcome these limitations and enhance scientific reasoning. Therefore, this position paper argues that MLLMs can significantly advance scientific reasoning across disciplines such as mathematics, physics, chemistry, and biology. First, we propose a four-stage research roadmap of scientific reasoning capabilities, and highlight the current state of MLLM applications in scientific reasoning, noting their ability to integrate and reason over diverse data types. Second, we summarize the key challenges that remain obstacles to achieving MLLM's full potential. To address these challenges, we propose actionable insights and suggestions for the future. Overall, our work offers a novel perspective on MLLM integration with scientific reasoning, providing the LLM community with a valuable vision for achieving Artificial General Intelligence (AGI).
ChatSR: Multimodal Large Language Models for Scientific Formula Discovery
AnnLab, Institute of Semiconductors, Chinese Academy of Sciences, Beijing, China(安 lab,半导体研究所,中国科学院,北京,中国);School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences, Beijing, China(电子、电气与通信工程学院,中国科学院大学,北京,中国);Zhongguancun Academy, Beijing, China(中关村学院,北京,中国);School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing 101408, China(先进交叉科学学院,中国科学院大学,北京101408,中国);College of Materials Science and Opto-Electronic Technology, University of Chinese Academy of Sciences, Beijing, 100049, China(材料科学与光电技术学院,中国科学院大学,北京100049,中国);School of Integrated Circuits, University of Chinese Academy of Sciences, Beijing 100049, China(集成电路学院,中国科学院大学,北京100049,中国)
Benchmarking Multimodal LLMs on Recognition and Understanding over Chemical Tables
State Key Laboratory of Cognitive Intelligence(认知智能国家重点实验室);University of Science and Technology of China(中国科学技术大学);Artificial Intelligence Research Institute(人工智能研究院);iFLYTEK Co., Ltd(iFLYTEK公司)
A unified multimodal understanding and generation model for cross-disciplinary scientific research
Fudan University(复旦大学);Shanghai Academy of Artificial Intelligence for Science(上海人工智能科学研究院);Shanghai Innovation Institute(上海创新研究院);FuXi Intelligent Computing Technology Co., Ltd.(伏羲智能计算科技有限公司)
ScImage: How Good Are Multimodal Large Language Models at Scientific Text-to-Image Generation?
University of Twente(特文特大学);University of Technology Nuremberg (UTN)(纽伦堡工业大学(UTN));University of Sheffield(谢菲尔德大学);Harbin Institute of Technology(哈尔滨工业大学);University of Mannheim(曼海姆大学);Technische Universität Dresden(德累斯顿工业大学)
Table Understanding and (Multimodal) LLMs: A Cross-Domain Case Study on Scientific vs. Non-Scientific Data
Deutsches Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)(德国人工智能研究中心有限公司);Technische Universität Berlin(柏林技术大学);BIFOLD;Deutsche Telekom(德国电信);Common Crawl Foundation(Common Crawl基金会);Humboldt-Universität zu Berlin(柏林洪堡大学)
Draw with Thought: Unleashing Multimodal Reasoning for Scientific Diagram Generation
Nanjing University of Information Science \& Technology Nanjing China;East China Normal University Shanghai China;The Hong Kong University of Science;Brown University Providence America;Southwest Jiaotong University Chengdu China;Nanjing University of Information Science \& Technology;East China Normal University;Brown University;Southwest Jiaotong University
提出无需训练的框架Draw with Thought,通过认知思维链推理将科学图表重建为可编辑的mxGraph XML代码,并发布Plot2XML基准,实验证明该方法能生成高保真且结构有效的重建结果。
A Pathway to General-Purpose Scientific AI: Multimodal Comprehension of Scientific Images
TIB Leibniz Information Centre for Science and Technology(莱布尼茨科学与技术信息中心(TIB));Eindhoven University of Technology(埃因霍温理工大学);Freie Universität Berlin(柏林自由大学);International Center for Chemical and Biological Sciences, University of Karachi(卡拉奇大学国际化学与生物科学中心);National University of Sciences and Technology(国家科技大学);University of Warwick(华威大学);PSG College of Technology(PSG理工学院);Aalto University(阿尔托大学)
MSEarth: A Multimodal Scientific Dataset and Benchmark for Phenomena Uncovering in Earth Science
The Hong Kong Polytechnic University(香港理工大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);The Chinese University of Hong Kong(香港中文大学)
Innovator-VL: A Multimodal Large Language Model for Scientific Discovery
School of Artificial Intelligence, Shanghai Jiao Tong University(上海交通大学人工智能学院);Institute of Theoretical Physics, Chinese Academy of Sciences(中国科学院理论物理研究所)
MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Research
Stanford University(斯坦福大学);Tsinghua University(清华大学);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Princeton University(普林斯顿大学);KTH Royal Institute of Technology(瑞典皇家理工学院);Chan Zuckerberg Biohub Network(陈·扎克翰生物中心网络)
Multimodal Deep Learning for Scientific Imaging Interpretation
Science and Technology Division, Corning Incorporated(康宁公司科技事业部);Robert Frederick Smith School of Chemical and Biomolecular Engineering, Cornell University(康奈尔大学罗伯特·弗雷德里克·史密斯化学与生物分子工程学院)
本文提出多模态深度学习框架 GlassLLaVA,结合论文图文数据与 GPT-4,实现玻璃材料 SEM 图像的类人解释、特征识别和缺陷检测,并建立可对照研究答案的评估指标。
Querying Multimodal Scientific Papers with AI: Practices and Preferences Across Blind, Low-Vision, and Sighted Scientists
Paul G. Allen School of Computer Science \& Engineering University of Washington Seattle WA USA;The Information School University of Washington Allen Institute for AI Seattle WA USA;Allen Institute for AI Seattle WA USA;University of Washington;Allen Institute for AI
DocHop-QA: Towards Multi-Hop Reasoning over Multimodal Document Collections
Pohang University of Science and Technology(釜山科学技术大学);The University of Sydney(悉尼大学);The University of Western Australia(西澳大学);The University of Melbourne(墨尔本大学)
ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows
The University of Hong Kong(香港大学);Shanghai AI Laboratory(上海人工智能实验室);Fudan University(复旦大学);Peking University(北京大学);Nanjing University(南京大学);East China Normal University(华东师范大学);Yale University(耶鲁大学)
SciOrch: Learning to Orchestrate Expert LLMs for Solving Frontier Multimodal Scientific Reasoning Tasks
Imperial College London(伦敦帝国学院);The Chinese University of Hong Kong(香港中文大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Shanghai Jiao Tong University(上海交通大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);University of Oxford(牛津大学);Shenzhen Loop Area Institute(深圳河套学院)