Q&A Prompts：通过挖掘问答提示发现丰富视觉线索，用于需要多样世界知识的VQA
Q&A Prompts: Discovering Rich Visual Clues through Mining Question-Answer Prompts for VQA requiring Diverse World Knowledge
AI总结：
本文提出Q&A Prompts方法，通过训练视觉问题生成模型挖掘图像中的问答对作为丰富视觉线索，并将其作为提示输入多模态大语言模型，在OK-VQA和A-OKVQA等需要世界知识的视觉问答数据集上取得显著提升。
摘要：
随着多模态大语言模型的突破，回答需要高级推理能力和世界知识的复杂视觉问题，已成为比以往任何时候都更重要的开发AI模型的测试平台。然而，由于人类的认知机制尚未被系统理解，为AI模型配备稳健的跨模态推理能力仍然具有挑战性。在本文中，我们认为如果能尽可能多地收集给定图像中的视觉线索，我们将更准确地识别图像，更好地理解问题，更轻松地回忆相关知识，并最终推理出答案。我们通过挖掘图像中的问答对，并将它们作为提示发送到多模态大语言模型中来发现这些丰富的视觉线索。我们将提出的方法称为Q&A Prompts。具体而言，我们首先使用训练集中的图像-答案对和相应的问题作为输入和输出，训练一个视觉问题生成模型。然后，我们使用图像标签模型识别各种实例，并将打包好的图像-标签对发送到视觉问题生成模型中，以提取的图像标签为答案生成相关问题。最后，我们使用视觉感知提示模块将这些生成的问答对编码为提示，并将它们发送到预训练的多模态大语言模型中，以推理出最终答案。实验结果表明，与最先进的方法相比，我们的Q&A Prompts在需要对多样世界知识进行推理的具有挑战性的视觉问答数据集（例如OK-VQA和A-OKVQA）上取得了实质性改进。
英文摘要：
With the breakthrough of multi-modal large language models, answering complex visual questions that demand advanced reasoning abilities and world knowledge has become a much more important testbed for developing AI models than ever. However, equipping AI models with robust cross-modality reasoning ability remains challenging since the cognition scheme of humans has not been understood systematically. In this paper, we believe that if we can collect visual clues in the given image as much as possible, we will recognize the image more accurately, understand the question better, recall relevant knowledge more easily, and finally reason out the answer. We discover these rich visual clues by mining question-answer pairs in images and sending them into multi-modal large language models as prompts. We call the proposed method Q&A Prompts. Specifically, we first use the image-answer pairs and the corresponding questions in the training set as inputs and outputs to train a visual question generation model. Then, we use an image tagging model to identify various instances and send packaged image-tag pairs into the visual question generation model to generate relevant questions with the extracted image tags as answers. Finally, we encode these generated question-answer pairs as prompts with a visual-aware prompting module and send them into pre-trained multi-modal large language models to reason out the final answers. Experimental results show that, compared with state-of-the-art methods, our Q&A Prompts achieves substantial improvements on the challenging visual question answering datasets requiring reasoning over diverse world knowledge, such as OK-VQA and A-OKVQA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
