arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.12801cs.CV
收藏

RelationVLM:使大型视觉-语言模型理解视觉关系

RelationVLM: Making Large Vision-Language Models Understand Visual Relations

University of Science and Technology of China(中国科学技术大学);Microsoft Research Asia(微软亚洲研究院)

Zhipeng Huang, Zhizheng Zhang, Zheng-Jun Zha, Yan Lu, Baining Guo

AI总结:

RelationVLM通过多阶段关系感知训练和数据策略,使大型视觉-语言模型能理解跨图像和视频的语义、时间及几何关系,增强少样本推理能力。

摘要:

大型视觉-语言模型(LVLMs)的发展正努力追赶大型语言模型(LLMs)的成功,但仍面临更多待解决的挑战。近期的工作使LVLMs能够定位对象级视觉内容并将其与文本进行对应。然而,由于缺乏相关数据,当前的LVLMs在精确理解视觉关系方面仍存在困难。在这项工作中,我们提出了RelationVLM,一种能够理解跨多张图像或视频内各种级别和类型关系的大型视觉-语言模型。具体而言,我们设计了一种多阶段关系感知训练方案以及一系列相应的数据配置策略,赋予RelationVLM理解语义关系、时间关联和几何变换的能力。广泛案例研究和定量评估表明,RelationVLM在理解此类关系方面具有强大能力,并通过比较展现出从少样本示例中进行推理的令人印象深刻的情境内能力。这项工作通过使LVLMs能够支持更广泛的下游应用,推动其向通用人工智能迈进。

英文摘要:

The development of Large Vision-Language Models (LVLMs) is striving to catch up with the success of Large Language Models (LLMs), yet it faces more challenges to be resolved. Very recent works enable LVLMs to localize object-level visual contents and ground text to them. Nonetheless, current LVLMs still struggle to precisely understand visual relations due to the lack of relevant data. In this work, we present RelationVLM, a large vision-language model capable of comprehending various levels and types of relations whether across multiple images or within a video. Specifically, we devise a multi-stage relation-aware training scheme and a series of corresponding data configuration strategies to bestow RelationVLM with the capabilities of understanding semantic relations, temporal associations and geometric transforms. Extensive case studies and quantitative evaluations show RelationVLM has strong capability in understanding such relations and emerges impressive in-context capability of reasoning from few-shot examples by comparison. This work fosters the advancements of LVLMs by enabling them to support a wider range of downstream applications toward artificial general intelligence.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑