理解 ME？面向细粒度视觉常识的多模态评估
Understanding ME? Multimodal Evaluation for Fine-grained Visual Commonsense
AI总结：
针对视觉语言模型在视觉常识理解中评估数据有限的问题，提出多模态评估（ME）流程自动生成问答对，并证明使用ME数据训练可提升标准VCR评估性能，同时揭示了低层信息辅助高层信息学习及视觉信息利用不足等发现。
摘要：
视觉常识理解要求视觉语言（VL）模型不仅要理解图像和文本，还需在两者间交叉引用以全面整合并实现对所描述视觉场景的理解。近期已开发出多种方法并在视觉常识基准上取得高性能。然而，由于评估数据资源有限，尚不清楚模型是否真正理解了视觉场景及潜在的常识知识。为提供深入分析，我们提出一种多模态评估（ME）流程，自动生成问答对以测试模型对视觉场景、文本及相关知识的理解。我们进一步证明，使用 ME 数据进行训练可提升模型在标准 VCR 评估中的性能。最后，我们的深入分析与比较揭示了有趣发现：（1）语义低层信息可辅助高层信息的学习，反之则不行；（2）与文本相比，视觉信息普遍存在利用不足。
英文摘要：
Visual commonsense understanding requires Vision Language (VL) models to not only understand image and text but also cross-reference in-between to fully integrate and achieve comprehension of the visual scene described. Recently, various approaches have been developed and have achieved high performance on visual commonsense benchmarks. However, it is unclear whether the models really understand the visual scene and underlying commonsense knowledge due to limited evaluation data resources. To provide an in-depth analysis, we present a Multimodal Evaluation (ME) pipeline to automatically generate question-answer pairs to test models' understanding of the visual scene, text, and related knowledge. We then take a step further to show that training with the ME data boosts the model's performance in standard VCR evaluation. Lastly, our in-depth analysis and comparison reveal interesting findings: (1) semantically low-level information can assist the learning of high-level information but not the opposite; (2) visual information is generally under utilization compared with text.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
