arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.13758cs.CVcs.AI
收藏

OmniScience: 一个大规模多模态数据集用于科学图像理解

OmniScience: A Large-scale Multi-modal Dataset for Scientific Image Understanding

DP Technology(DP技术)

Haoyi Tao, Chaozheng Huang, Nan Wang, Han Lyu, Linfeng Zhang, Guolin Ke, Xi Fang

AI总结:

OmniScience是一个大规模多模态数据集,通过动态模型路由生成高信息密度的图像标题,提升多模态模型在科学图像理解上的性能。

摘要:

多模态大语言模型在自然图像理解上表现出色,但在解释科学图像方面能力有限,包括但不限于示意图、实验表征和分析图表。这种局限性在开源MLLMs中尤为明显。这种差距主要源于现有数据集领域覆盖有限、结构标注粗略、语义关联薄弱。我们引入了OmniScience,一个包含150万个图-标题-上下文三元组的大规模高保真多模态数据集,覆盖超过10个主要科学领域。为了获得信息密度更高、准确性更高的图像标题数据以用于多模态大模型训练,我们开发了一种动态模型路由再标题生成流程,利用最先进的多模态大语言模型,通过联合合成视觉特征、原始图标题和相应文本参考(由科学家撰写的)生成密集且自包含的描述。该流程进一步通过严格的质量筛选和与人类专家判断对齐,确保事实准确性与语义完整性,并将图像-文本多模态相似度分数从0.769提升到0.956。我们进一步提出了一种标题问答协议作为评估视觉理解的代理任务。在此设置下,在OmniScience上微调的Qwen2.5-VL-3B模型在MM-MT-Bench上比基线高出0.378,在MMMU上高出0.140。

英文摘要:

Multimodal Large Language Models demonstrate strong performance on natural image understanding, yet exhibit limited capability in interpreting scientific images, including but not limited to schematic diagrams, experimental characterizations, and analytical charts. This limitation is particularly pronounced in open-source MLLMs. The gap largely stems from existing datasets with limited domain coverage, coarse structural annotations, and weak semantic grounding. We introduce OmniScience, a large-scale, high-fidelity multi-modal dataset comprising 1.5 million figure-caption-context triplets, spanning more than 10 major scientific disciplines. To obtain image caption data with higher information density and accuracy for multi-modal large-model training, we develop a dynamic model-routing re-captioning pipeline that leverages state-of-the-art multi-modal large language models to generate dense, self-contained descriptions by jointly synthesizing visual features, original figure captions, and corresponding in-text references authored by human scientists. The pipeline is further reinforced with rigorous quality filtering and alignment with human expert judgments, ensuring both factual accuracy and semantic completeness, and boosts the image-text multi-modal similarity score from 0.769 to 0.956. We further propose a caption QA protocol as a proxy task for evaluating visual understanding. Under this setting, Qwen2.5-VL-3B model finetuned on OmniScience show substantial gains over baselines, achieving a gain of 0.378 on MM-MT-Bench and a gain of 0.140 on MMMU.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑