基于语言模型反馈的零样本视觉问答
Zero-shot Visual Question Answering with Language Model Feedback
AI总结：
本文提出LAMOC方法，通过语言模型反馈引导captioning模型生成更符合任务目标的描述，在零样本知识型VQA中取得优于竞争方法的效果。
摘要：
在本文中，我们提出了一种新颖的语言模型引导的captioning方法LAMOC，用于基于知识的视觉问答（VQA）。我们的方法将由captioning模型生成的描述作为答案预测模型的上下文，该答案预测模型是一个预训练语言模型（PLM）。作为主要贡献，我们利用预测模型的引导和反馈来提升captioning模型的能力。通过这种方式，captioning模型能够感知来自PLM的任务目标和信息需求。为了开发我们的方法，我们设计了两个具体的训练阶段，其中第一阶段使captioning模型适应预测模型（为训练选择更合适的描述候选），第二阶段根据任务目标调整captioning模型（从PLM的反馈中学习）。大量实验证明了所提方法在基于知识的VQA任务上的有效性。具体而言，在具有挑战性的A-OKVQA数据集上，LAMOC优于多种有竞争力的零样本方法，甚至取得了与微调VLP模型相当的结果。我们的代码已公开发布在https://github.com/RUCAIBox/LAMOC。
英文摘要：
In this paper, we propose a novel language model guided captioning approach, LAMOC, for knowledge-based visual question answering (VQA). Our approach employs the generated captions by a captioning model as the context of an answer prediction model, which is a Pre-trained Language model (PLM). As the major contribution, we leverage the guidance and feedback of the prediction model to improve the capability of the captioning model. In this way, the captioning model can become aware of the task goal and information need from the PLM. To develop our approach, we design two specific training stages, where the first stage adapts the captioning model to the prediction model (selecting more suitable caption propositions for training) and the second stage tunes the captioning model according to the task goal (learning from feedback of the PLM). Extensive experiments demonstrate the effectiveness of the proposed approach on the knowledge-based VQA task. Specifically, on the challenging A-OKVQA dataset, LAMOC outperforms several competitive zero-shot methods and even achieves comparable results to a fine-tuned VLP model. Our code is publicly available at https://github.com/RUCAIBox/LAMOC.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
