arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.00231cs.CVcs.CL
收藏

Multimodal ArXiv:一个用于提升大型视觉语言模型科学理解能力的数据集

Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models

The University of Hong Kong(香港大学);Peking University(北京大学)

Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, Qi Liu

AI总结:

本文提出 Multimodal ArXiv 数据集(含 ArXivCap 和 ArXivQA),以提升大型视觉语言模型对科学图表的理解与数学推理能力,并揭示当前模型在学术图形语义理解上的不足。

摘要:

大型视觉语言模型(LVLMs)在涉及自然场景具体图像的多种任务中表现出色。然而,由于科学领域训练数据集的稀缺,它们解释抽象图形(如几何形状和科学图表)的能力仍然有限。为了填补这一空白,我们引入了 Multimodal ArXiv,包含 ArXivCap 和 ArXivQA,以增强 LVLMs 的科学理解能力。ArXivCap 是一个图像-标题数据集,包含 6.4M 张图像和 3.9M 个标题,来源于跨越各种科学领域的 572K 篇 ArXiv 论文。基于 ArXivCap,我们通过提示 GPT-4V 根据科学图表生成问答,引入了问答数据集 ArXivQA。ArXivQA 极大地增强了开源 LVLMs 的数学推理能力,在多模态数学推理基准上实现了 10.4% 的绝对准确率提升。此外,利用 ArXivCap,我们设计了四个视觉到文本任务来对 LVLMs 进行基准测试。使用最先进的 LVLMs 进行的评估结果突显了它们在处理学术图形细微语义时的困难,而特定领域的训练则带来了显著的性能提升。我们的错误分析揭示了当前 LVLMs 对视觉上下文的误解、识别错误以及生成过于简化的标题,为未来的改进提供了启示。

英文摘要:

Large vision-language models (LVLMs) excel across diverse tasks involving concrete images from natural scenes. However, their ability to interpret abstract figures, such as geometry shapes and scientific plots, remains limited due to a scarcity of training datasets in scientific domains. To fill this gap, we introduce Multimodal ArXiv, consisting of ArXivCap and ArXivQA, for enhancing LVLMs scientific comprehension. ArXivCap is a figure-caption dataset comprising 6.4M images and 3.9M captions, sourced from 572K ArXiv papers spanning various scientific domains. Drawing from ArXivCap, we introduce ArXivQA, a question-answering dataset generated by prompting GPT-4V based on scientific figures. ArXivQA greatly enhances open-sourced LVLMs' mathematical reasoning capabilities, achieving a 10.4\% absolute accuracy gain on a multimodal mathematical reasoning benchmark. Furthermore, employing ArXivCap, we devise four vision-to-text tasks for benchmarking LVLMs. Evaluation results with state-of-the-art LVLMs underscore their struggle with the nuanced semantics of academic figures, while domain-specific training yields substantial performance gains. Our error analysis uncovers misinterpretations of visual context, recognition errors, and the production of overly simplified captions by current LVLMs, shedding light on future improvements.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑