arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.00264cs.CV
收藏

S1-MMAlign: 一个大规模、跨学科的数据集用于科学图像-文本理解

S1-MMAlign: A Large-Scale, Multi-Disciplinary Dataset for Scientific Figure-Text Understanding

Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉学科学院);University of Chinese Academy of Sciences(中国科学院大学)

He Wang, Longteng Guo, Pengkang Huo, Xuanxu Lin, Yichen Yuan, Jie Jiang, Jing Liu

AI总结:

S1-MMAlign是一个大规模跨学科数据集,通过语义增强提升科学图像与文本的对齐质量,推动AI在科学领域的应用。

摘要:

多模态学习已革新了通用领域任务,但其在科学发现中的应用受到复杂科学图像与稀疏文本描述之间深刻语义差距的阻碍。我们提出了S1-MMAlign,一个大规模、跨学科的多模态数据集,包含来自250万篇开放获取科学论文的1550万对高质量图像-文本对。涵盖物理、生物、工程等学科,数据集捕捉了实验设置、热图和显微图像等多样化的视觉模态。为解决原始科学描述中普遍存在的对齐问题,我们引入了一种AI-ready的语义增强流程,利用Qwen-VL多模态大模型系列通过整合论文摘要和引用上下文的语境来重新描述图像。技术验证显示,这种增强显著提高了数据质量:基于SciBERT的伪困惑度度量显示减少了语义模糊性,而CLIP分数表明图像-文本对齐性提高了18.21%。S1-MMAlign为推进人工智能科学时代的科学推理和跨模态理解提供了基础资源。该数据集可在https://huggingface.co/datasets/ScienceOne-AI/S1-MMAlign公开获取。

英文摘要:

Multimodal learning has revolutionized general domain tasks, yet its application in scientific discovery is hindered by the profound semantic gap between complex scientific imagery and sparse textual descriptions. We present S1-MMAlign, a large-scale, multi-disciplinary multimodal dataset comprising over 15.5 million high-quality image-text pairs derived from 2.5 million open-access scientific papers. Spanning disciplines from physics and biology to engineering, the dataset captures diverse visual modalities including experimental setups, heatmaps, and microscopic imagery. To address the pervasive issue of weak alignment in raw scientific captions, we introduce an AI-ready semantic enhancement pipeline that utilizes the Qwen-VL multimodal large model series to recaption images by synthesizing context from paper abstracts and citation contexts. Technical validation demonstrates that this enhancement significantly improves data quality: SciBERT-based pseudo-perplexity metrics show reduced semantic ambiguity, while CLIP scores indicate an 18.21% improvement in image-text alignment. S1-MMAlign provides a foundational resource for advancing scientific reasoning and cross-modal understanding in the era of AI for Science. The dataset is publicly available at https://huggingface.co/datasets/ScienceOne-AI/S1-MMAlign.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑