PromptCap：提示引导的任务感知图像描述生成
PromptCap: Prompt-Guided Task-Aware Image Captioning
AI总结：
本文提出 PromptCap，通过问题提示生成任务感知图像描述，以连接图像与黑盒 GPT-3，并在知识型 VQA 上取得最优准确率。
摘要：
基于知识的视觉问答（VQA）涉及的问题，需要图像之外的世界知识才能得出正确答案。像 GPT-3 这样的大型语言模型（LM）凭借其强大的知识检索和推理能力，对该任务尤其有帮助。为了使 LM 能够理解图像，先前的工作使用描述生成模型将图像转换为文本。然而，当用单句描述概括一幅图像时，应当描述哪些视觉实体往往并不明确。通用图像描述经常会遗漏 LM 正确回答视觉问题所必需的视觉细节。为应对这一挑战，我们提出 PromptCap（Prompt-guided image Captioning，提示引导的图像描述生成），这是一个描述生成模型，旨在更好地充当图像与黑盒 LM 之间的连接器。与通用描述不同，PromptCap 接收自然语言提示，以控制生成描述中要描述的视觉实体。该提示包含描述应协助回答的问题。为避免额外标注，PromptCap 使用由 GPT-3 和现有数据集合成的示例进行训练。我们在一个既有流程上展示了 PromptCap 的有效性，在该流程中，GPT-3 以图像描述作为提示来执行 VQA。PromptCap 大幅优于通用描述，并在基于知识的 VQA 任务上取得最先进准确率（OK-VQA 上为 60.4%，A-OKVQA 上为 59.6%）。WebQA 上的零样本结果表明，PromptCap 能够很好地泛化到未见过的领域。
英文摘要：
Knowledge-based visual question answering (VQA) involves questions that require world knowledge beyond the image to yield the correct answer. Large language models (LMs) like GPT-3 are particularly helpful for this task because of their strong knowledge retrieval and reasoning capabilities. To enable LM to understand images, prior work uses a captioning model to convert images into text. However, when summarizing an image in a single caption sentence, which visual entities to describe are often underspecified. Generic image captions often miss visual details essential for the LM to answer visual questions correctly. To address this challenge, we propose PromptCap (Prompt-guided image Captioning), a captioning model designed to serve as a better connector between images and black-box LMs. Different from generic captions, PromptCap takes a natural-language prompt to control the visual entities to describe in the generated caption. The prompt contains a question that the caption should aid in answering. To avoid extra annotation, PromptCap is trained by examples synthesized with GPT-3 and existing datasets. We demonstrate PromptCap's effectiveness on an existing pipeline in which GPT-3 is prompted with image captions to carry out VQA. PromptCap outperforms generic captions by a large margin and achieves state-of-the-art accuracy on knowledge-based VQA tasks (60.4% on OK-VQA and 59.6% on A-OKVQA). Zero-shot results on WebQA show that PromptCap generalizes well to unseen domains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
