MEBench：一种新的评估视觉-语言模型互斥偏差的基准
MEBench: A Novel Benchmark for Understanding Mutual Exclusivity Bias in Vision-Language Models
AI总结：
MEBench通过引入空间推理提升评估难度，评估视觉-语言模型对互斥偏差的处理能力，发现模型存在弱互斥偏差但能利用额外空间上下文解决歧义。
摘要：
本文介绍MEBench，一种新的评估视觉-语言模型互斥偏差的基准。不同于传统ME任务，MEBench进一步融入空间推理以创建更具挑战性和现实感的评估环境。为便于受控实验，我们还提出了一个灵活且可扩展的数据生成管道，支持构建多样化的标注场景。我们使用新的评估指标来评估各种视觉-语言模型（VLMs）在该基准上的性能。我们发现这些VLMs表现出较弱的互斥偏差，但能够在多个新颖物体设置中利用额外的空间上下文来解决歧义。
英文摘要：
This paper introduces MEBench, a novel benchmark for evaluating mutual exclusivity (ME) bias, a cognitive phenomenon observed in children during word learning. Unlike traditional ME tasks, MEBench further incorporates spatial reasoning to create more challenging and realistic evaluation settings. To facilitate controlled experimentation, we also present a flexible and scalable data generation pipeline that supports the construction of diverse annotated scenes. We assess the performance of various vision-language models (VLMs) on this benchmark using novel evaluation metrics that capture key aspects of ME-based reasoning. We find that these VLMs exhibit weak ME bias, while showing some ability to leverage extra spatial context to resolve ambiguity in multiple novel object settings. Project page: http://mebench.github.io/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
