arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.11248cs.CV
收藏

CoLLaVO:Crayon 大语言与视觉模型

CoLLaVO: Crayon Large Language and Vision mOdel

KAIST(韩国科学技术院)

Byung-Kwan Lee, Beomchan Park, Chae Won Kim, Yong Man Ro

AI总结:

本文提出 CoLLaVO 模型,通过引入基于全景颜色图的 Crayon Prompt 视觉提示微调方案和 Dual QLoRA 学习策略,有效增强了视觉语言模型的物体级图像理解能力,并在零样本设置下的多个视觉语言基准上取得显著提升。

摘要:

大语言模型(LLMs)和指令微调的显著成功推动了视觉语言模型(VLMs)向多功能通用模型的演进。然而,当前的 VLMs 是否真正具备由“图像中有哪些物体?”或“哪个物体对应于指定的边界框?”所决定的优质物体级图像理解能力,仍有待探索。我们的发现表明,当前 VLMs 的图像理解能力与其在视觉语言(VL)任务上的零样本性能高度相关。这意味着,优先考虑基础图像理解对于 VLMs 在 VL 任务中表现出色至关重要。为了增强物体级图像理解,我们提出了 Crayon 大语言与视觉模型(CoLLaVO),该模型引入了基于全景颜色图的 Crayon Prompt 作为新的视觉提示微调方案进行指令微调。此外,我们提出了一种 Dual QLoRA 学习策略,以在视觉指令微调过程中保留物体级图像理解而不遗忘,从而在零样本设置下的众多 VL 基准上实现了显著飞跃。

英文摘要:

The remarkable success of Large Language Models (LLMs) and instruction tuning drives the evolution of Vision Language Models (VLMs) towards a versatile general-purpose model. Yet, it remains unexplored whether current VLMs genuinely possess quality object-level image understanding capabilities determined from 'what objects are in the image?' or 'which object corresponds to a specified bounding box?'. Our findings reveal that the image understanding capabilities of current VLMs are strongly correlated with their zero-shot performance on vision language (VL) tasks. This suggests that prioritizing basic image understanding is crucial for VLMs to excel at VL tasks. To enhance object-level image understanding, we propose Crayon Large Language and Vision mOdel (CoLLaVO), which incorporates instruction tuning with Crayon Prompt as a new visual prompt tuning scheme based on panoptic color maps. Furthermore, we present a learning strategy of Dual QLoRA to preserve object-level image understanding without forgetting it during visual instruction tuning, thereby achieving a significant leap in numerous VL benchmarks in a zero-shot setting.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑