Ref-Adv：探索提及表达任务中的多模态大语言模型视觉推理
Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks
AI总结：
Ref-Adv 是一个旨在评估多模态大语言模型视觉推理能力的基准，通过增加表达复杂性和减少干扰项，揭示模型在视觉推理和 grounding 方面的不足。
摘要：
提及表达理解（REC）将语言与区域级视觉感知联系起来。标准基准（RefCOCO、RefCOCO+、RefCOCOg）在多模态大语言模型（LLMs）的推动下迅速发展，但仍然是视觉推理和 grounding 的薄弱测试：（i）许多表达非常简短，留下很少的推理需求；（ii）图像通常包含很少的干扰项，使目标容易找到；（iii）冗余描述符允许捷径解决方案，绕过真正的文本理解和视觉推理。我们引入了 Ref-Adv，一个现代 REC 基准，通过将语言上非平凡的表达与仅包含识别目标所需信息的配对来抑制捷径。该数据集包含在真实图像上的提及表达，通过硬干扰项进行筛选，并用包含否定的推理方面进行注释。我们进行了全面的消融实验（词序扰动和描述符删除充分性）以显示解决 Ref-Adv 需要超越简单提示的推理，并在 Ref-Adv 上评估了广泛的现代多模态 LLMs。尽管在 RefCOCO、RefCOCO+ 和 RefCOCOg 上表现强劲，模型在 Ref-Adv 上大幅下降，揭示了对捷径的依赖以及视觉推理和 grounding 的差距。我们提供了深入的失败分析，并希望 Ref-Adv 能指导未来多模态 LLMs 中的视觉推理和 grounding 研究。
英文摘要：
Referring Expression Comprehension (REC) links language to region level visual perception. Standard benchmarks (RefCOCO, RefCOCO+, RefCOCOg) have progressed rapidly with multimodal LLMs but remain weak tests of visual reasoning and grounding: (i) many expressions are very short, leaving little reasoning demand; (ii) images often contain few distractors, making the target easy to find; and (iii) redundant descriptors enable shortcut solutions that bypass genuine text understanding and visual reasoning. We introduce Ref-Adv, a modern REC benchmark that suppresses shortcuts by pairing linguistically nontrivial expressions with only the information necessary to uniquely identify the target. The dataset contains referring expressions on real images, curated with hard distractors and annotated with reasoning facets including negation. We conduct comprehensive ablations (word order perturbations and descriptor deletion sufficiency) to show that solving Ref-Adv requires reasoning beyond simple cues, and we evaluate a broad suite of contemporary multimodal LLMs on Ref-Adv. Despite strong results on RefCOCO, RefCOCO+, and RefCOCOg, models drop markedly on Ref-Adv, revealing reliance on shortcuts and gaps in visual reasoning and grounding. We provide an in depth failure analysis and aim for Ref-Adv to guide future work on visual reasoning and grounding in MLLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
