arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.01280cs.CVcs.AIcs.CL
收藏

二次审视:多模态大语言模型中的免训练证据高亮

Look Twice: Training-Free Evidence Highlighting for Knowledge-based Visual Question Answering

University of Modena and Reggio Emilia(摩德纳和雷焦艾米利亚大学)

Marco Morini, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara

AI总结:

本文提出Look Twice框架,通过模型注意力模式识别相关视觉区域和文本证据,提升预训练MLLMs在多模态证据利用中的表现,实验显示在多个VQA基准上效果显著。

摘要:

回答图像相关问题通常需要结合视觉理解与外部知识。多模态大语言模型(MLLMs)为此提供了自然框架,但它们在回答知识密集型查询时往往难以识别最相关的视觉和文本证据。在这样的场景中,模型必须整合视觉线索与检索到的文本证据(通常是嘈杂或部分相关的),同时在图像中定位细粒度的视觉信息。在本工作中,我们引入Look Twice(LoT),一种无需训练的推理时间框架,旨在提升预训练MLLMs对多模态证据的利用能力。具体而言,我们利用模型的注意力模式来估计哪些视觉区域和检索到的文本元素与查询相关,并基于此高亮证据生成答案。所选线索通过轻量级提示级标记进行高亮,鼓励模型在生成过程中重新关注相关证据。在多个基于知识的VQA基准上的实验显示,相对于零样本MLLMs,效果有明显提升。此外,在视觉导向和幻觉导向的基准上的进一步评估表明,仅使用视觉证据高亮即可在没有文本上下文的设置中提升模型性能,且无需额外训练或架构修改。源代码将公开发布。

英文摘要:

Knowledge-based Visual Question Answering (KB-VQA) requires Multimodal Large Language Models (MLLMs) to identify and combine fine-grained visual cues with retrieved textual evidence. However, retrieval often introduces noisy and partially relevant content, while images contain distracting visual regions, causing pretrained MLLMs to overlook the evidence that actually supports the answer. To address this, we introduce Look Twice (LoT), a training-free inference-time framework that turns the model's own internal attention into an explicit multimodal evidence-selection mechanism. LoT first leverages the model's internal attention patterns to identify query-relevant image regions and textual sentences, filters attention sinks and distracting content, and reformulates the input to explicitly highlight the selected evidence before answer generation. The method requires no parameter updates, auxiliary models, or architectural modifications. Across four KB-VQA benchmarks and ten off-the-shelf MLLMs ranging from 2B to 38B parameters, LoT improves every evaluated backbone, with average gains of up to +12.5 accuracy points. It also provides further gains when combined with established context-refinement strategies, yielding additional improvements over already refined inputs. These results establish LoT as a general and effective mechanism for enabling pretrained MLLMs to exploit available multimodal evidence more accurately. Source code is publicly available at https://aimagelab.github.io/LoT/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑