CLIP-UP: 基于CLIP的视觉问答中不可回答问题检测
CLIP-UP: CLIP-Based Unanswerable Problem Detection for Visual Question Answering
AI总结：
CLIP-UP通过基于CLIP的相似性度量，为视觉问答模型提供检测不可回答问题的能力，提升模型在不可回答问题上的识别性能。
摘要：
视觉-语言模型（VLMs）在视觉理解和推理方面表现出显著的能力，例如在视觉问答（VQA）中，模型被要求回答与视觉输入相关的疑问。然而，这些模型仍可能产生明显不自然的错误，例如提供（错误的）答案来回答不可回答的VQA问题，例如询问图像中未出现的物体。为了解决这一问题，我们提出了CLIP-UP：基于CLIP的不可回答问题检测，一种新颖的轻量方法，用于为VLMs赋予能够避免回答不可回答问题的能力。CLIP-UP利用基于CLIP的相似性度量来提取问题-图像对齐信息以检测不可回答性，仅需要高效训练少量额外层，同时保持原始VLMs的权重不变。在多个模型上测试后，CLIP-UP在评估不可回答性的多个选择和开放性VQA基准上取得了显著改进，超越了其他方法，同时在其他任务上保持原始性能。
英文摘要：
Vision-Language Models (VLMs) demonstrate remarkable capabilities in visual understanding and reasoning, such as in Visual Question Answering (VQA), where the model is asked a question related to a visual input. Still, these models can make distinctly unnatural errors, for example, providing (wrong) answers to unanswerable VQA questions, such as questions asking about objects that do not appear in the image. To address this issue, we propose CLIP-UP: CLIP-based Unanswerable Problem detection, a novel lightweight method for equipping VLMs with the ability to withhold answers to unanswerable questions. CLIP-UP leverages CLIP-based similarity measures to extract question-image alignment information to detect unanswerability, requiring efficient training of only a few additional layers, while keeping the original VLMs' weights unchanged. Tested across several models, CLIP-UP achieves significant improvements on benchmarks assessing unanswerability in both multiple-choice and open-ended VQA, surpassing other methods, while preserving original performance on other tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
