arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.19934cs.CLcs.AI
收藏

从最少到最多:通过数据合成构建一个即插即用的视觉推理器

From the Least to the Most: Building a Plug-and-Play Visual Reasoner via Data Synthesis

Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Ant Group(蚂蚁集团)

Chuanqi Cheng, Jian Guan, Wei Wu, Rui Yan

AI总结:

针对视觉语言模型多步推理数据缺乏的问题,提出从最少到最多的视觉推理范式及数据合成方法,构建5万样本微调出即插即用的视觉推理器,在四个VQA基准上显著提升了四种VLMs的推理能力。

摘要:

我们探索了视觉语言模型(VLMs)中的多步推理。该问题具有挑战性,因为由多步视觉和语言处理组成的推理数据几乎不可用。为了克服这一挑战,我们首先引入了一种从最少到最多的视觉推理范式,该范式将问题分解为子问题的步骤与调用外部工具解决子问题的步骤交替进行。基于该范式,我们进一步提出了一种新颖的数据合成方法,能够以自底向上的方式自动为图像创建问题和多步推理路径。我们的方法将复杂的合成任务划分为几个简单的子任务,并(几乎完全)依赖开源模型来完成这些子任务。因此,整个合成过程是可复现且成本高效的,并且合成数据的质量得到保证。通过该方法,我们构建了5万个视觉推理样本。随后,我们通过监督微调开发了一个视觉推理器,它能够以即插即用的方式普遍增强各种现有VLMs的推理能力。大量实验表明,该视觉推理器在四个VQA基准上能够持续且显著地提升四种VLMs的性能。我们的代码和数据集可在 https://github.com/steven-ccq/VisualReasoner 获取。

英文摘要:

We explore multi-step reasoning in vision-language models (VLMs). The problem is challenging, as reasoning data consisting of multiple steps of visual and language processing are barely available. To overcome the challenge, we first introduce a least-to-most visual reasoning paradigm, which interleaves steps of decomposing a question into sub-questions and invoking external tools for resolving sub-questions. Based on the paradigm, we further propose a novel data synthesis approach that can automatically create questions and multi-step reasoning paths for an image in a bottom-up manner. Our approach divides the complex synthesis task into a few simple sub-tasks, and (almost entirely) relies on open-sourced models to accomplish the sub-tasks. Therefore, the entire synthesis process is reproducible and cost-efficient, and the synthesized data is quality guaranteed. With the approach, we construct $50$k visual reasoning examples. Then, we develop a visual reasoner through supervised fine-tuning, which is capable of generally enhancing the reasoning abilities of a wide range of existing VLMs in a plug-and-play fashion. Extensive experiments indicate that the visual reasoner can consistently and significantly improve four VLMs on four VQA benchmarks. Our code and dataset are available at https://github.com/steven-ccq/VisualReasoner.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑