学习纠错：面向视觉常识推理干扰项的可解释反馈生成
Learning to Correction: Explainable Feedback Generation for Visual Commonsense Reasoning Distractor
AI总结：
本研究针对大语言多模态模型视觉常识推理干扰项纠错能力不足的问题，构建了可解释反馈基准数据集VCR-DF，并提出教学专家指令反馈生成模型PEIFG，性能显著优于现有模型，为相关能力评估提供了新方向。
摘要：
大语言多模态模型（LMMs）在视觉常识推理（VCR）任务中已展现出优异性能，该任务旨在基于图像内的视觉常识回答选择题。然而，LMMs在干扰项中出现潜在视觉常识错误时对其进行纠正的能力仍未得到充分探究。受人类教师设计有难度的干扰项以测试学生对概念或技能的理解，并协助学生识别并纠正错误以得出正确答案的方式启发，我们是首个让LMMs模拟这一纠错过程的研究。为此，我们采用GPT-4作为“教师”，收集了用于纠错的可解释反馈数据集VCR-DF，该数据集作为基准，用于评估LMMs识别VCR干扰项中的错误认知并阐明错误背后原因以得出最终答案的能力。此外，我们提出了一种基于LMM的教学专家指令反馈生成（PEIFG）模型，将可学习的专家提示与多模态指令结合，作为反馈生成的引导。实验结果表明，我们的PEIFG显著优于现有LMMs。我们认为该基准为评估LMMs的能力提供了新方向。
英文摘要：
Large multimodal models (LMMs) have shown remarkable performance in the visual commonsense reasoning (VCR) task, which aims to answer a multiple-choice question based on visual commonsense within an image. However, the ability of LMMs to correct potential visual commonsense errors in the distractor upon their occurrence is yet under-explored. Drawing inspiration from how a human teacher crafts challenging distractors to test students' comprehension of the concepts or skills and assists them in identifying and correcting errors toward the answer, we are the pioneering research for LMMs to simulate this error correction process. To this end, we employ GPT-4 as a ``teacher'' to collect the explainable feedback dataset VCR-DF for error correction, which serves as a benchmark to evaluate the ability of LMMs to identify misconceptions and clarify reasons behind the error in VCR distractors toward final answers. In addition, we propose an LMM-based Pedagogical Expert Instructed Feedback Generation (PEIFG) model to incorporate the learnable expert prompts and multimodal instruction as guidance for feedback generation. Experimental results show that our PEIFG significantly outperforms existing LMMs. We believe that our benchmark provides a new direction for evaluating the capabilities of LMMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
