超越视觉相似度：面向基于知识的视觉问答的实体对齐检索
Beyond Visual Similarity: Entity-Aligned Retrieval for Knowledge-Based Visual Question Answering
查看机构详情
- Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）
- University of Arizona（亚利桑那大学）
机构由 AI 辅助整理，请以论文原文为准。
浏览论文内容
中文总结 AI 辅助
针对KB-VQA现有检索范式忽略实体语义对齐的缺陷，提出首个MLLM嵌入检索器KBMR，结合语义判别器与连续语义蒸馏目标，在Recall@1和VQA准确率上较CLIP基线实现显著提升。
中文摘要 AI 辅助
基于知识的视觉问答（KB-VQA）依赖于检索外部信息来回答涉及长尾实体的查询。然而，现有的检索流程大多采用CLIP风格的双编码器，这类编码器优先考虑表面层面的视觉相似度，而非实体层面的语义对齐。当语义相同的概念呈现出较大的视觉差异，或不同实体在视觉上相似时，这种范式往往会失效。为解决这一问题，我们提出KBMR，这是首个专为KB-VQA定制的基于多模态大语言模型（MLLM）的嵌入检索器。KBMR利用MLLM强大的自回归能力，将图像映射到能更好保留概念身份的语义空间。为应对维基百科规模检索中的噪声监督挑战，我们引入了一个基于MLLM的语义判别器，该判别器会生成连续的实体一致性权重。这些权重引导一种新颖的连续语义蒸馏目标，支持有效的难负样本采样和超越刚性二元标签的软监督。大量实验表明，KBMR的性能显著优于CLIP基线，在检索Recall@1上的提升最高达14.7%，在端到端VQA准确率上的提升达9.4%。代码可在指定的URL获取。
英文摘要
Knowledge-Based Visual Question Answering (KB-VQA) relies on retrieving external information to answer queries involving long-tail entities. However, existing retrieval pipelines predominantly employ CLIP-style dual encoders, which prioritize surface-level visual similarity over entity-level semantic alignment. This paradigm often fails when semantically identical concepts exhibit large visual variations or when distinct entities appear visually similar. To address this, we propose KBMR, the first MLLM-based embedding retriever tailored for KB-VQA. Leveraging the robust autoregressive capabilities of MLLMs, KBMR maps images into a semantic space that better preserves concept identity. To tackle the challenge of noisy supervision in Wikipedia-scale retrieval, we introduce an MLLM-based semantic discriminator that generates continuous entity-consistency weights. These weights guide a novel continuous semantic distillation objective, enabling effective hard negative sampling and soft supervision beyond rigid binary labels. Extensive experiments demonstrate that KBMR significantly outperforms CLIP baselines, yielding up to a 14.7% improvement in retrieval Recall@1 and a 9.4% gain in end-to-end VQA accuracy. Code is available at https://github.com/realHarryX/KBMR.
