Large vision-language models (LVLMs) excel across diverse tasks involving concrete images from natural scenes. However, their ability to interpret abstract figures, such as geometry shapes and scientific plots, remains limited due to a scarcity of training datasets in scientific domains. To fill this gap, we introduce Multimodal ArXiv, consisting of ArXivCap and ArXivQA, for enhancing LVLMs scientific comprehension. ArXivCap is a figure-caption dataset comprising 6.4M images and 3.9M captions, sourced from 572K ArXiv papers spanning various scientific domains. Drawing from ArXivCap, we introduce ArXivQA, a question-answering dataset generated by prompting GPT-4V based on scientific figures. ArXivQA greatly enhances open-sourced LVLMs' mathematical reasoning capabilities, achieving a 10.4\% absolute accuracy gain on a multimodal mathematical reasoning benchmark. Furthermore, employing ArXivCap, we devise four vision-to-text tasks for benchmarking LVLMs. Evaluation results with state-of-the-art LVLMs underscore their struggle with the nuanced semantics of academic figures, while domain-specific training yields substantial performance gains. Our error analysis uncovers misinterpretations of visual context, recognition errors, and the production of overly simplified captions by current LVLMs, shedding light on future improvements.
A Pathway to General-Purpose Scientific AI: Multimodal Comprehension of Scientific Images
TIB Leibniz Information Centre for Science and Technology(莱布尼茨科学与技术信息中心(TIB));Eindhoven University of Technology(埃因霍温理工大学);Freie Universität Berlin(柏林自由大学);International Center for Chemical and Biological Sciences, University of Karachi(卡拉奇大学国际化学与生物科学中心);National University of Sciences and Technology(国家科技大学);University of Warwick(华威大学);PSG College of Technology(PSG理工学院);Aalto University(阿尔托大学)
ScImage: How Good Are Multimodal Large Language Models at Scientific Text-to-Image Generation?
University of Twente(特文特大学);University of Technology Nuremberg (UTN)(纽伦堡工业大学(UTN));University of Sheffield(谢菲尔德大学);Harbin Institute of Technology(哈尔滨工业大学);University of Mannheim(曼海姆大学);Technische Universität Dresden(德累斯顿工业大学)
SciEducator: Scientific Video Understanding and Educating via Deming-Cycle Multi-Agent System
Jinan University(济南大学);National University of Singapore(新加坡国立大学);Nanyang Technological University(南洋理工大学);Peking University(北京大学);University of Electronic Science and Technology of China(电子科技大学);South China University of Technology(华南理工大学);Guangming Laboratory(光明实验室);Zhejiang University(浙江大学)
Multimodal Deep Learning for Scientific Imaging Interpretation
Science and Technology Division, Corning Incorporated(康宁公司科技事业部);Robert Frederick Smith School of Chemical and Biomolecular Engineering, Cornell University(康奈尔大学罗伯特·弗雷德里克·史密斯化学与生物分子工程学院)
本文提出多模态深度学习框架 GlassLLaVA,结合论文图文数据与 GPT-4,实现玻璃材料 SEM 图像的类人解释、特征识别和缺陷检测,并建立可对照研究答案的评估指标。
SciVQR: A Multidisciplinary Multimodal Benchmark for Advanced Scientific Reasoning Evaluation
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);OPPO AI Center(OPPO AI中心)