arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.05472cs.CV
收藏

面向多模态推理的问题感知视觉变换器

Question Aware Vision Transformer for Multimodal Reasoning

Technion(以色列理工学院);AWS AI Labs(亚马逊云科技人工智能实验室)

Roy Ganz, Yair Kittenplon, Aviad Aberdam, Elad Ben Avraham, Oren Nuriel, Shai Mazor, Ron Litman

AI总结:

提出QA-ViT,一种将问题感知嵌入视觉编码器的模型无关方法,动态聚焦图像相关区域,在多种多模态架构上实现一致性能提升。

摘要:

视觉-语言(VL)模型已获得显著的研究关注,推动了多模态推理领域的重大进展。这些架构通常包含一个视觉编码器、一个大语言模型(LLM)以及一个将视觉特征与LLM表示空间对齐的投影模块。尽管取得了成功,一个关键局限仍然存在:视觉编码过程与用户查询(通常以图像相关问题形式出现)脱节。因此,生成的视觉特征可能无法最优地适应图像中与查询相关的元素。为解决这一问题,我们提出了QA-ViT,一种用于多模态推理的问题感知视觉变换器方法,该方法将问题感知直接嵌入视觉编码器中。这种集成产生了动态的视觉特征,聚焦于图像中与所提问题相关的方面。QA-ViT是模型无关的,可以高效地集成到任何VL架构中。大量实验证明了我们将该方法应用于多种多模态架构的有效性,在多样化任务上带来了一致的改进,展示了其在增强视觉和场景文本理解方面的潜力。

英文摘要:

Vision-Language (VL) models have gained significant research focus, enabling remarkable advances in multimodal reasoning. These architectures typically comprise a vision encoder, a Large Language Model (LLM), and a projection module that aligns visual features with the LLM's representation space. Despite their success, a critical limitation persists: the vision encoding process remains decoupled from user queries, often in the form of image-related questions. Consequently, the resulting visual features may not be optimally attuned to the query-specific elements of the image. To address this, we introduce QA-ViT, a Question Aware Vision Transformer approach for multimodal reasoning, which embeds question awareness directly within the vision encoder. This integration results in dynamic visual features focusing on relevant image aspects to the posed question. QA-ViT is model-agnostic and can be incorporated efficiently into any VL architecture. Extensive experiments demonstrate the effectiveness of applying our method to various multimodal architectures, leading to consistent improvement across diverse tasks and showcasing its potential for enhancing visual and scene-text understanding.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑