Multimodal Large Language Models demonstrate strong performance on natural image understanding, yet exhibit limited capability in interpreting scientific images, including but not limited to schematic diagrams, experimental characterizations, and analytical charts. This limitation is particularly pronounced in open-source MLLMs. The gap largely stems from existing datasets with limited domain coverage, coarse structural annotations, and weak semantic grounding. We introduce OmniScience, a large-scale, high-fidelity multi-modal dataset comprising 1.5 million figure-caption-context triplets, spanning more than 10 major scientific disciplines. To obtain image caption data with higher information density and accuracy for multi-modal large-model training, we develop a dynamic model-routing re-captioning pipeline that leverages state-of-the-art multi-modal large language models to generate dense, self-contained descriptions by jointly synthesizing visual features, original figure captions, and corresponding in-text references authored by human scientists. The pipeline is further reinforced with rigorous quality filtering and alignment with human expert judgments, ensuring both factual accuracy and semantic completeness, and boosts the image-text multi-modal similarity score from 0.769 to 0.956. We further propose a caption QA protocol as a proxy task for evaluating visual understanding. Under this setting, Qwen2.5-VL-3B model finetuned on OmniScience show substantial gains over baselines, achieving a gain of 0.378 on MM-MT-Bench and a gain of 0.140 on MMMU.
Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams
Nanjing University of Science and Technology(南京理工大学);Baidu Inc(百度公司);AIML, Adelaide University(阿德莱德大学AIML);SUTD(新加坡科技设计大学);Southeast University(东南大学);East China Normal University(华东师范大学);University of Oxford(牛津大学)
ChatSR: Multimodal Large Language Models for Scientific Formula Discovery
AnnLab, Institute of Semiconductors, Chinese Academy of Sciences, Beijing, China(安 lab,半导体研究所,中国科学院,北京,中国);School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences, Beijing, China(电子、电气与通信工程学院,中国科学院大学,北京,中国);Zhongguancun Academy, Beijing, China(中关村学院,北京,中国);School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing 101408, China(先进交叉科学学院,中国科学院大学,北京101408,中国);College of Materials Science and Opto-Electronic Technology, University of Chinese Academy of Sciences, Beijing, 100049, China(材料科学与光电技术学院,中国科学院大学,北京100049,中国);School of Integrated Circuits, University of Chinese Academy of Sciences, Beijing 100049, China(集成电路学院,中国科学院大学,北京100049,中国)
LeMat-Synth: a multi-modal toolbox to curate broad synthesis procedure databases from scientific literature
Massachusetts Institute of Technology(麻省理工学院);Entalpic(Entalpic公司);URM MAP;LIRIS(LIRIS实验室);Georgia Tech(佐治亚理工学院);University of California Santa Barbara(加州大学圣巴巴拉分校);Chalmers University of Technology(查尔姆斯理工大学);University of Gothenburg(哥德堡大学);ETH Zurich(苏黎世联邦理工学院);EPFL(洛桑联邦理工学院);Johns Hopkins University(约翰斯·霍普金斯大学);Hugging Face(Hugging Face公司);Imperial College London(帝国理工学院)
A unified multimodal understanding and generation model for cross-disciplinary scientific research
Fudan University(复旦大学);Shanghai Academy of Artificial Intelligence for Science(上海人工智能科学研究院);Shanghai Innovation Institute(上海创新研究院);FuXi Intelligent Computing Technology Co., Ltd.(伏羲智能计算科技有限公司)
SciCoQA: Quality Assurance for Scientific Paper--Code Alignment
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, TU Darmstadt and National Research Center for Applied Cybersecurity ATHENE(普遍知识处理实验室(UKP实验室)、计算机科学系、德累斯顿技术大学和应用网络安全国家研究中心ATHENE)
SciTS: Scientific Time Series Understanding and Generation with LLMs
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Tsinghua University(清华大学);Harbin Institute of Technology(哈尔滨工业大学);University of Science and Technology of China(中国科学技术大学)
InVitroVision: a Multi-Modal AI Model for Automated Description of Embryo Development using Natural Language
Software Competence Center Hagenberg GmbH(软件能力中心海根贝格有限公司);Kinderwunsch Zentrum Kepler Universitätsklinikum(儿童愿望中心凯普勒大学诊所);Wunschkind Klinik Dr. Brunbauer(愿望宝宝诊所布兰鲍尔医生)
xVal: A Continuous Numerical Tokenization for Scientific Language Models
Flatiron Institute(弗莱蒂伦研究所);Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室);University of Cambridge(剑桥大学);Université Paris-Saclay(巴黎萨克雷大学);Université Paris Cité(巴黎西岱大学);CEA(法国原子能和替代能源委员会);CNRS(法国国家科学研究中心);University of Colorado Boulder(科罗拉多大学博尔德分校);New York University(纽约大学);Prescient Design;Genentech(基因泰克公司);Princeton University(普林斯顿大学)
Rethinking Domain Specialization for Open-Ended Scientific Reasoning in Astronomy Language Models
Oak Ridge National Laboratory(橡树岭国家实验室);The Ohio State University(俄亥俄州立大学);QUP/IPNS, High Energy Accelerator Research Organization (KEK)(高能加速器研究机构(KEK)QUP/IPNS);Max-Planck-Institut für Astronomie(马克斯·普朗克天文研究所)
A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language
Renmin University of China(中国人民大学);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);University of Chinese Academy of Sciences(中国科学院大学);The Chinese University of Hong Kong(香港中文大学)
MatterChat: A Multi-Modal LLM for Material Science
Applied Mathematics and Computational Research Division, Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室应用数学与计算研究部);National Energy Research Scientific Computing Center, Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室国家能源研究科学计算中心);NSF National AI Institute for Student-AI Teaming, University of Colorado at Boulder(国家科学基金会学生-人工智能协同人工智能研究所,科罗拉多大学波德罗分校);Department of Electrical and Computer Engineering, The University of Utah(犹他大学电气与计算机工程系);Scientific Data Division, Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室科学数据部);International Computer Science Institute, Berkeley, CA, USA(伯克利加州大学国际计算机科学研究所);Department of Statistics, University of California at Berkeley(伯克利加州大学统计学系)
SciMMIR: Benchmarking Scientific Multi-modal Information Retrieval
Multimodal Art Projection Research Community(多模态艺术投影研究共同体);University of Manchester(曼彻斯特大学);University of Waterloo(滑铁卢大学);Hong Kong University of Science and Technology(香港科技大学);Durham University(杜伦大学)
S1-MMAlign: A Large-Scale, Multi-Disciplinary Dataset for Scientific Figure-Text Understanding
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉学科学院);University of Chinese Academy of Sciences(中国科学院大学)
GAIA: A Global, Multi-modal, Multi-scale Vision-Language Dataset for Remote Sensing Image Analysis
Orion Lab, School of Rural, Surveying and Geoinformatics Engineering, National Technical University of Athens(奥里昂实验室,农村测绘与地理信息工程学院,希腊雅典国家技术大学);Institute of Astronomy, Astrophysics, Space Applications and Remote Sensing, National Observatory of Athens(天文、天体物理、空间应用与遥感研究所,希腊雅典国家天文台);Department of Informatics and Telematics, Harokopio University of Athens(信息与电信技术系,雅典霍罗科皮欧大学);Chair of Data Science in Earth Observation, Technical University of Munich(地球观测数据科学教授职位,慕尼黑技术大学);Munich Center for Machine Learning(慕尼黑机器学习中心);Faculty of Electrical Engineering and Computer Science, Technische Universität Berlin(电气工程与计算机科学系,柏林技术大学);BIFOLD - Berlin Institute for the Foundations of Learning and Data(柏林学习与数据基础研究所)