CIVET：视觉语言模型理解能力的系统性评估
CIVET: Systematic Evaluation of Understanding in VLMs
AI总结：
本文提出CIVET框架，通过受控刺激系统性评估视觉语言模型对物体属性与关系的理解，发现现有模型识别能力有限、受位置影响大且关系理解薄弱，未达人类水平。
摘要：
尽管视觉语言模型（VLMs）在各种任务中已取得具有竞争力的性能，但它们对场景底层结构和语义的理解仍研究不足。为了探究VLMs的理解能力，我们以受控且可解释的方式研究它们对物体属性和关系的处理能力。为此，我们提出了CIVET，一个新颖且可扩展的系统性评估框架，通过受控刺激进行系统评估。CIVET解决了缺乏标准化系统性评估VLMs理解能力的问题，使研究人员能够以统计严谨性检验假设。利用CIVET，我们在无注释噪声、无数据集特定偏差、无不受控场景复杂性的详尽刺激集上评估了五个最先进的VLMs。我们的发现表明：1）当前VLMs仅能准确识别有限的一组基本物体属性；2）它们的性能在很大程度上依赖于物体在场景中的位置；3）它们难以理解物体间的基本关系。此外，与人类标注者的对比评估显示，VLMs仍未能达到人类水平的准确性。
英文摘要：
While Vision-Language Models (VLMs) have achieved competitive performance in various tasks, their comprehension of the underlying structure and semantics of a scene remains understudied. To investigate the understanding of VLMs, we study their capability regarding object properties and relations in a controlled and interpretable manner. To this scope, we introduce CIVET, a novel and extensible framework for systematiC evaluatIon Via controllEd sTimuli. CIVET addresses the lack of standardized systematic evaluation for assessing VLMs' understanding, enabling researchers to test hypotheses with statistical rigor. With CIVET, we evaluate five state-of-the-art VLMs on exhaustive sets of stimuli, free from annotation noise, dataset-specific biases, and uncontrolled scene complexity. Our findings reveal that 1) current VLMs can accurately recognize only a limited set of basic object properties; 2) their performance heavily depends on the position of the object in the scene; 3) they struggle to understand basic relations among objects. Furthermore, a comparative evaluation with human annotators reveals that VLMs still fall short of achieving human-level accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
