CrossCheck-Bench：多模态冲突解决中的组成性故障诊断
CrossCheck-Bench: Diagnosing Compositional Failures in Multimodal Conflict Resolution
AI总结：
CrossCheck-Bench通过多阶段注释流程构建的基准测试，揭示了多模态模型在处理跨模态冲突时的瓶颈，并表明融合符号推理与视觉处理的方法能提升模型的稳健性。
摘要：
多模态大语言模型主要是在对齐的图像-文本对上进行训练和评估，这使得它们检测和解决现实世界不一致性的能力大多未被探索。在开放领域应用中，视觉和文本线索经常冲突，要求模型进行超越表面匹配的结构化推理。我们介绍了CrossCheck-Bench，一个用于评估多模态输入矛盾检测的诊断基准。该基准采用一个分层任务框架，涵盖三个层次的推理复杂性，并定义了七种解决跨模态不一致性的基本能力。CrossCheck-Bench包含15000个问题-答案对，这些对来源于现实世界物品，并通过合成注入矛盾进行处理。该数据集通过一个多阶段注释流程构建，涉及超过450小时专家时间，以确保语义有效性以及感知、整合和推理方面的校准难度。我们评估了13种最先进的视觉-语言模型，并观察到当任务从感知匹配转向逻辑矛盾检测时，性能持续下降。大多数模型在孤立实体识别方面表现良好，但在需要综合多个线索进行冲突推理时却失败。能力层面分析进一步揭示了技能获取的不均衡，尤其是在需要多步骤推理或基于规则验证的任务中。此外，进一步的探测显示，传统的提示策略如链式思维和标记集仅能带来微小的提升。相比之下，将符号推理与基于现实的视觉处理相结合的方法能够实现更稳定的提升。这些结果突显了多模态推理中的持续瓶颈，并提出了构建能够稳健跨模态验证的模型的新方向。
英文摘要：
Multimodal Large Language Models are primarily trained and evaluated on aligned image-text pairs, which leaves their ability to detect and resolve real-world inconsistencies largely unexplored. In open-domain applications visual and textual cues often conflict, requiring models to perform structured reasoning beyond surface-level alignment. We introduce CrossCheck-Bench, a diagnostic benchmark for evaluating contradiction detection in multimodal inputs. The benchmark adopts a hierarchical task framework covering three levels of reasoning complexity and defines seven atomic capabilities essential for resolving cross-modal inconsistencies. CrossCheck-Bench includes 15k question-answer pairs sourced from real-world artifacts with synthetically injected contradictions. The dataset is constructed through a multi-stage annotation pipeline involving more than 450 expert hours to ensure semantic validity and calibrated difficulty across perception, integration, and reasoning. We evaluate 13 state-of-the-art vision-language models and observe a consistent performance drop as tasks shift from perceptual matching to logical contradiction detection. Most models perform well on isolated entity recognition but fail when multiple clues must be synthesized for conflict reasoning. Capability-level analysis further reveals uneven skill acquisition, especially in tasks requiring multi-step inference or rule-based validation. Additional probing shows that conventional prompting strategies such as Chain-of-Thought and Set-of-Mark yield only marginal gains. By contrast, methods that interleave symbolic reasoning with grounded visual processing achieve more stable improvements. These results highlight a persistent bottleneck in multimodal reasoning and suggest new directions for building models capable of robust cross-modal verification.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
