探究零样本与少样本视觉问答中的提示技术
Investigating Prompting Techniques for Zero- and Few-Shot Visual Question Answering
AI总结：
本文研究视觉语言模型在零样本与少样本视觉问答中的提示策略，发现问题模板选择、图像描述增强、自一致性推理及LLM引导预处理能有效提升模型性能。
摘要：
在本文中，我们探索了有效的提示技术，以提升当代视觉语言模型（VLMs）在零样本和少样本视觉问答（VQA）中的表现。我们研究的核心是问题模板在引导VLMs生成准确答案方面的作用。我们发现特定模板会显著影响VQA结果，这凸显了策略性选择模板的必要性。我们研究的另一个关键方面是用图像描述来增强VLMs，在VQA任务中为它们提供直接图像特征之外的额外视觉线索。令人惊讶的是，尽管VLMs能够直接“看到”图像，这种增强在许多情况下仍显著提升了它们的性能！我们探索了思维链（CoT）推理，并发现虽然标准的CoT推理会导致性能下降，但诸如自一致性（self-consistency）之类的先进方法可以帮助恢复性能。此外，我们发现仅文本的少样本示例能增强VLMs与任务格式的对齐，尤其有利于那些倾向于生成冗长零样本答案的模型。最后，为了缓解使用基于字符串匹配的VQA指标评估自由形式开放式VQA回答时面临的挑战，我们引入了一种简单的LLM引导的预处理技术，以使模型回答适应预期的真实答案分布。总之，我们的研究揭示了VLMs在VQA中提示策略的复杂性，强调了描述、模板和预处理之间的协同使用以提升模型效能。
英文摘要：
In this paper, we explore effective prompting techniques to enhance zero- and few-shot Visual Question Answering (VQA) performance in contemporary Vision-Language Models (VLMs). Central to our investigation is the role of question templates in guiding VLMs to generate accurate answers. We identify that specific templates significantly influence VQA outcomes, underscoring the need for strategic template selection. Another pivotal aspect of our study is augmenting VLMs with image captions, providing them with additional visual cues alongside direct image features in VQA tasks. Surprisingly, this augmentation significantly improves the VLMs' performance in many cases, even though VLMs "see" the image directly! We explore chain-of-thought (CoT) reasoning and find that while standard CoT reasoning causes drops in performance, advanced methods like self-consistency can help recover it. Furthermore, we find that text-only few-shot examples enhance VLMs' alignment with the task format, particularly benefiting models prone to verbose zero-shot answers. Lastly, to mitigate the challenges associated with evaluating free-form open-ended VQA responses using string-matching based VQA metrics, we introduce a straightforward LLM-guided pre-processing technique to adapt the model responses to the expected ground-truth answer distribution. In summary, our research sheds light on the intricacies of prompting strategies in VLMs for VQA, emphasizing the synergistic use of captions, templates, and pre-processing to enhance model efficacy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
