GC-KBVQA：一种增强基于知识的视觉问答性能的新型四阶段框架
GC-KBVQA: A New Four-Stage Framework for Enhancing Knowledge Based Visual Question Answering Performance
AI总结：
提出GC-KBVQA四阶段框架，利用接地字幕和外部知识增强LLM的零样本视觉问答性能，无需微调，显著优于现有方法。
摘要：
基于知识的视觉问答（KB-VQA）方法专注于需要利用图像中明确内容之外的信息进行推理的任务。早期方法依赖显式知识库来提供辅助信息。近期方法利用大型语言模型（LLMs）作为隐式知识源。尽管KB-VQA方法已展现出有前景的结果，但其潜力仍受限于所提供的辅助文本可能与问题上下文不相关，且可能包含无关信息，从而误导答案预测器。我们提出了一种名为“接地字幕引导的基于知识的视觉问答”（GC-KBVQA）的新型四阶段框架，使LLMs能够有效执行零样本VQA任务，而无需端到端的多模态训练。创新之处包括：接地的问题感知字幕生成，以超越通用描述，提供紧凑且详细、上下文丰富的信息。该信息与外部来源的知识相结合，为LLM创建信息量丰富的提示。GC-KBVQA可处理多种VQA任务，且无需任务特定的微调，从而通过利用通用的预训练LLMs降低成本和部署复杂性。与竞争性KB-VQA方法的比较显示性能显著提升。我们的代码将公开。
英文摘要：
Knowledge-Based Visual Question Answering (KB-VQA) methods focus on tasks that demand reasoning with information extending beyond the explicit content depicted in the image. Early methods relied on explicit knowledge bases to provide this auxiliary information. Recent approaches leverage Large Language Models (LLMs) as implicit knowledge sources. While KB-VQA methods have demonstrated promising results, their potential remains constrained as the auxiliary text provided may not be relevant to the question context, and may also include irrelevant information that could misguide the answer predictor. We introduce a novel four-stage framework called Grounding Caption-Guided Knowledge-Based Visual Question Answering (GC-KBVQA), which enables LLMs to effectively perform zero-shot VQA tasks without the need for end-to-end multimodal training. Innovations include grounding question-aware caption generation to move beyond generic descriptions and have compact, yet detailed and context-rich information. This is combined with knowledge from external sources to create highly informative prompts for the LLM. GC-KBVQA can address a variety of VQA tasks, and does not require task-specific fine-tuning, thus reducing both costs and deployment complexity by leveraging general-purpose, pre-trained LLMs. Comparison with competing KB-VQA methods shows significantly improved performance. Our code will be made public.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
