通过问题驱动的图像描述作为提示增强视觉问答
Enhancing Visual Question Answering through Question-Driven Image Captions as Prompts
AI总结：
针对零样本视觉问答的挑战，提出将问题驱动的图像描述作为提示融入大语言模型的方法，通过提取问题关键词生成定制描述，在GQA基准零样本设置下取得有竞争力的性能。
摘要：
视觉问答（VQA）被认为是一项AI完备任务，因为它需要对视觉和语言内容进行理解、推理与推断。过去几年，人们提出了大量用于解决VQA问题的神经网络架构。然而，零样本VQA的实现仍是一项挑战，因为它需要高级的泛化与推理能力。本研究探讨了在VQA流程中引入图像描述作为中间步骤的影响。具体而言，我们探究了用图像描述替代图像，并利用大语言模型（LLM）构建零样本设置的有效性。由于图像描述是该流程中最关键的步骤，我们从结构和语义层面，比较了前沿图像描述模型对不同问题类型下VQA性能的影响。我们在该流程中提出了一种简洁高效的问题驱动图像描述方法，用于将上下文信息传递给问答（QA）模型。该方法包括从问题中提取关键词，利用这些关键词为每个图像-问题对生成描述，并将问题驱动的描述融入LLM提示中。我们评估了在VQA流程中使用通用图像描述与问题驱动图像描述的效果。本研究凸显了利用图像描述并借助LLM能力，在零样本设置下于GQA基准上取得有竞争力性能的潜力。我们的代码可在https://github.com/ovguyo/captions-in-VQA获取。
英文摘要：
Visual question answering (VQA) is known as an AI-complete task as it requires understanding, reasoning, and inferring about the vision and the language content. Over the past few years, numerous neural architectures have been suggested for the VQA problem. However, achieving success in zero-shot VQA remains a challenge due to its requirement for advanced generalization and reasoning skills. This study explores the impact of incorporating image captioning as an intermediary process within the VQA pipeline. Specifically, we explore the efficacy of utilizing image captions instead of images and leveraging large language models (LLMs) to establish a zero-shot setting. Since image captioning is the most crucial step in this process, we compare the impact of state-of-the-art image captioning models on VQA performance across various question types in terms of structure and semantics. We propose a straightforward and efficient question-driven image captioning approach within this pipeline to transfer contextual information into the question-answering (QA) model. This method involves extracting keywords from the question, generating a caption for each image-question pair using the keywords, and incorporating the question-driven caption into the LLM prompt. We evaluate the efficacy of using general-purpose and question-driven image captions in the VQA pipeline. Our study highlights the potential of employing image captions and harnessing the capabilities of LLMs to achieve competitive performance on GQA under the zero-shot setting. Our code is available at \url{https://github.com/ovguyo/captions-in-VQA}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
