arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.17971cs.CVcs.AIcs.CL
收藏

聚合图像中的一切:面向图像内学习的方法

All in an Aggregated Image for In-Image Learning

Singapore Management University(新加坡管理大学);Southwest Jiaotong University(西南交通大学);Singapore University of Technology and Design(新加坡科技设计大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))

Lei Wang, Wanyu Xu, Zhiqiang Hu, Yihuai Lan, Shan Dong, Hao Wang, Roy Ka-Wei Lee, Ee-Peng Lim

AI总结:

本文提出图像内学习(I²L)机制,将演示示例、视觉线索和思维链整合到单张聚合图像中,以提升大型多模态模型的推理能力,并进一步提出自动选择 I²L 或其他 ICL 方法的 I²L-Hybrid 策略。

摘要:

本文提出了一种新的上下文学习(ICL)机制,称为图像内学习(I²L),它将演示示例、视觉线索和思维链推理整合到一张聚合图像中,以增强大型多模态模型(如 GPT-4V)在多模态推理任务中的能力。与以往依赖将图像转换为文本或将视觉输入融入语言模型的方法不同,I²L 将所有信息整合到一张聚合图像中,并利用图像处理、理解和推理能力。这具有若干优势:它减少了对复杂图像的不准确文本描述,提供了在定位演示示例时的灵活性,并避免了多张输入图像和冗长的提示。我们还提出了 I²L-Hybrid,这是一种将 I²L 的优势与其他 ICL 方法相结合的方法。具体而言,它采用自动策略为特定任务实例选择最合适的方法(I²L 或另一种特定的 ICL 方法)。我们进行了大量实验,以评估 I²L 和 I²L-Hybrid 在 MathVista 上的有效性,该基准涵盖了各种复杂的多模态推理任务。此外,我们还研究了图像分辨率、单张图像中演示示例的数量以及这些演示在聚合图像中的位置对 I²L 有效性的影响。我们的代码已在 https://github.com/AGI-Edgerunners/IIL 上公开。

英文摘要:

This paper introduces a new in-context learning (ICL) mechanism called In-Image Learning (I$^2$L) that combines demonstration examples, visual cues, and chain-of-thought reasoning into an aggregated image to enhance the capabilities of Large Multimodal Models (e.g., GPT-4V) in multimodal reasoning tasks. Unlike previous approaches that rely on converting images to text or incorporating visual input into language models, I$^2$L consolidates all information into an aggregated image and leverages image processing, understanding, and reasoning abilities. This has several advantages: it reduces inaccurate textual descriptions of complex images, provides flexibility in positioning demonstration examples, and avoids multiple input images and lengthy prompts. We also introduce I$^2$L-Hybrid, a method that combines the strengths of I$^2$L with other ICL methods. Specifically, it uses an automatic strategy to select the most suitable method (I$^2$L or another certain ICL method) for a specific task instance. We conduct extensive experiments to assess the effectiveness of I$^2$L and I$^2$L-Hybrid on MathVista, which covers a variety of complex multimodal reasoning tasks. Additionally, we investigate the influence of image resolution, the number of demonstration examples in a single image, and the positions of these demonstrations in the aggregated image on the effectiveness of I$^2$L. Our code is publicly available at https://github.com/AGI-Edgerunners/IIL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑