GPT-4V在知识密集型视觉问答上的全面评估
A Comprehensive Evaluation of GPT-4V on Knowledge-Intensive Visual Question Answering
AI总结：
本文从常识知识、细粒度世界知识和具有决策依据的综合知识三个视角全面评估了GPT-4V等MLMs在知识密集型VQA任务上的能力，发现其在处理世界知识时存在严重幻觉，并验证了视觉知识增强训练与提示技术的改进潜力。
摘要：
多模态大模型（MLMs）的出现显著推动了视觉理解领域的发展，在视觉问答（VQA）领域展现出卓越的能力。然而，然而，真正的挑战在于知识密集型VQA任务，这类任务不仅需要仅需要识别视觉元素，还需要结合庞大的已学知识库对视觉信息进行深度理解。为了揭示MLMs的此类能力，特别是新推出的GPT-4V和Gemini，我们，从三个视角提供了深入评估：1）常识知识，评估模型理解视觉线索并联系一般知识的能力；2）细粒度世界知识，测试模型从图像中推理出特定知识的能力，展示其在各种专门领域的熟练程度；3）具有决策依据的综合知识，检查模型为推理提供逻辑解释的能力，从可解释性视角促进更深入的分析析。此外，我们利用视觉知识增强的训练策略和多模态检索增强生成方法来增强MLMs，突出了这一研究方向未来进展的必要性。大量实验表明：a）当使用复合图像作为少样本时，GPT-4V展现出增强的解释生成能力；b）在处理世界知识时，GPT-4V和其他MLMs会产生严重的幻觉；c）视觉知识增强的训练和提示技术展现出提升性能的潜力。代码：https://github.com/HITsz-TMG/Cognitive-Visual-Language-Mapper
英文摘要：
The emergence of multimodal large models (MLMs) has significantly advanced the field of visual understanding, offering remarkable capabilities in the realm of visual question answering (VQA). Yet, the true challenge lies in the domain of knowledge-intensive VQA tasks, which necessitate not just recognition of visual elements, but also a deep comprehension of the visual information in conjunction with a vast repository of learned knowledge. To uncover such capabilities of MLMs, particularly the newly introduced GPT-4V and Gemini, we provide an in-depth evaluation from three perspectives: 1) Commonsense Knowledge, which assesses how well models can understand visual cues and connect to general knowledge; 2) Fine-grained World Knowledge, which tests the model's skill in reasoning out specific knowledge from images, showcasing their proficiency across various specialized fields; 3) Comprehensive Knowledge with Decision-making Rationales, which examines model's capability to provide logical explanations for its inference, facilitating a deeper analysis from the interpretability perspective. Additionally, we utilize a visual knowledge-enhanced training strategy and multimodal retrieval-augmented generation approach to enhance MLMs, highlighting the future need for advancements in this research direction. Extensive experiments indicate that: a) GPT-4V demonstrates enhanced explanation generation when using composite images as few-shots; b) GPT-4V and other MLMs produce severe hallucinations when dealing with world knowledge; c) Visual knowledge enhanced training and prompting technicals present potential to improve performance. Codes: https://github.com/HITsz-TMG/Cognitive-Visual-Language-Mapper
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
