arXiv 2303.019032025-04-29cs.CVcs.CLcs.LG

Prophet：利用互补答案启发式提示大型语言模型进行基于知识的视觉问答

Prophet: Prompting Large Language Models with Complementary Answer Heuristics for Knowledge-based Visual Question Answering

Key Laboratory of Complex Systems Modeling and Simulation, the School of Computer Science, Hangzhou Dianzi University（复杂系统建模与仿真重点实验室、计算机科学学院、杭州电子大学）；HDU-ITMO Joint Institute, Hangzhou Dianzi University（杭州电子大学-ITMO联合学院）；School of Computer Science and Information Engineering, Hefei University of Technology（计算机科学与信息工程学院、合肥工业大学）；School of Intelligence Science and Engineering, Harbin Institute of Technology, Shenzhen（智能科学与工程学院、哈尔滨工业大学深圳校区）

Zhou Yu, Xuecheng Ouyang, Zhenwei Shao, Meng Wang, Jun Yu

备注：: An extended journal version of our CVPR 2023 paper, which has been accepted at IEEE T-PAMI 2025. The original conference version can be referred to as the v1 version
DOI：: 10.1109/TPAMI.2025.3562422

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出Prophet框架，通过从普通VQA模型提取答案候选和答案感知示例作为互补启发式，格式化提示LLM以增强其对图像和问题的理解，显著提升基于知识的VQA性能。

摘要：

基于知识的视觉问答（VQA）需要图像之外的外部知识来回答问题。早期研究从显式知识库（KBs）中检索所需知识，这常引入与问题无关的信息，从而限制了模型的性能。近期工作转而使用强大的大型语言模型（LLM）作为隐式知识引擎来获取回答所需的知识。尽管这些方法取得了令人鼓舞的结果，但我们认为它们并未完全激活盲LLM的能力，因为提供的文本输入不足以描绘回答问题所需的视觉信息。在本文中，我们提出了Prophet——一个概念简单、灵活且通用的框架，旨在用答案启发式提示LLM以进行基于知识的VQA。具体而言，我们首先在特定的基于知识的VQA数据集上训练一个不含外部知识的普通VQA模型。之后，我们从该VQA模型中提取两种互补的答案启发式：答案候选和答案感知示例。这两类答案启发式被联合编码进格式化的提示中，以促进LLM对图像和问题的理解，从而生成更准确的答案。通过结合最先进的LLM GPT-3，Prophet在四个具有挑战性的基于知识的VQA数据集上显著优于现有的最先进方法。Prophet具有通用性，可由不同的VQA模型（即判别式和生成式模型）与不同的LLM（即商业和开源模型）组合实例化。此外，Prophet还可以在不同阶段与现代大型多模态模型集成，称为Prophet++，以进一步提升在基于知识的VQA任务上的能力。

英文摘要：

Knowledge-based visual question answering (VQA) requires external knowledge beyond the image to answer the question. Early studies retrieve required knowledge from explicit knowledge bases (KBs), which often introduces irrelevant information to the question, hence restricting the performance of their models. Recent works have resorted to using a powerful large language model (LLM) as an implicit knowledge engine to acquire the necessary knowledge for answering. Despite the encouraging results achieved by these methods, we argue that they have not fully activated the capacity of the \emph{blind} LLM as the provided textual input is insufficient to depict the required visual information to answer the question. In this paper, we present Prophet -- a conceptually simple, flexible, and general framework designed to prompt LLM with answer heuristics for knowledge-based VQA. Specifically, we first train a vanilla VQA model on a specific knowledge-based VQA dataset without external knowledge. After that, we extract two types of complementary answer heuristics from the VQA model: answer candidates and answer-aware examples. The two types of answer heuristics are jointly encoded into a formatted prompt to facilitate the LLM's understanding of both the image and question, thus generating a more accurate answer. By incorporating the state-of-the-art LLM GPT-3, Prophet significantly outperforms existing state-of-the-art methods on four challenging knowledge-based VQA datasets. Prophet is general that can be instantiated with the combinations of different VQA models (i.e., both discriminative and generative ones) and different LLMs (i.e., both commercial and open-source ones). Moreover, Prophet can also be integrated with modern large multimodal models in different stages, which is named Prophet++, to further improve the capabilities on knowledge-based VQA tasks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。