BloomVQA：评估分层多模态理解能力
BloomVQA: Assessing Hierarchical Multi-modal Comprehension
AI总结：
本文提出BloomVQA数据集，基于Bloom分类法构建分层理解评估，揭示现有多模态模型在高层认知任务上准确率显著下降且存在视觉输入绕过等问题。
摘要：
我们提出了一个新的VQA数据集BloomVQA，以促进对大型视觉语言模型在理解任务上的全面评估。与当前通常侧重于基于事实的记忆和缺乏理论基础的简单推理任务的基准不同，我们基于图画故事收集多项选择样本，这些样本反映了不同层次的理解水平，其层次划分依据教育研究中广泛采用的经典学习评估框架——Bloom分类法。我们的数据映射到一种新颖的分层图表示，该表示能够实现自动数据增强，并支持刻画模型一致性的新度量。我们对近期多模态模型进行了分级评估和可靠性分析。与低层次任务相比，我们观察到模型在需要高级理解和认知技能的任务上表现下降，VQA准确率下降幅度高达38.0%。与早期模型相比，GPT-4V在所有理解层次上都表现出更高的准确率，并显示出绕过视觉输入的倾向，尤其是在高层次任务中。当前模型还在多种场景中表现出与人类理解不一致的模式，表明需要基于有理论依据的标准进行改进。
英文摘要：
We propose a novel VQA dataset, BloomVQA, to facilitate comprehensive evaluation of large vision-language models on comprehension tasks. Unlike current benchmarks that often focus on fact-based memorization and simple reasoning tasks without theoretical grounding, we collect multiple-choice samples based on picture stories that reflect different levels of comprehension, as laid out in Bloom's Taxonomy, a classic framework for learning assessment widely adopted in education research. Our data maps to a novel hierarchical graph representation which enables automatic data augmentation and novel measures characterizing model consistency. We perform graded evaluation and reliability analysis on recent multi-modal models. In comparison to low-level tasks, we observe decreased performance on tasks requiring advanced comprehension and cognitive skills with up to 38.0\% drop in VQA accuracy. In comparison to earlier models, GPT-4V demonstrates improved accuracy over all comprehension levels and shows a tendency of bypassing visual inputs especially for higher-level tasks. Current models also show consistency patterns misaligned with human comprehension in various scenarios, demonstrating the need for improvement based on theoretically-grounded criteria.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
