SEAM：用于视觉-语言模型的跨模态语义等价基准
SEAM: Semantically Equivalent Across Modalities Benchmark for Vision-Language Models
AI总结：
针对视觉-语言模型跨表征推理一致性评估的挑战，提出SEAM基准，通过跨模态语义等价输入配对，揭示模型模态失衡问题，为改进模态无关推理提供受控评估设置。
摘要：
评估视觉-语言模型（VLMs）在不同表征间的推理一致性具有挑战性，因为模态比较通常会被任务差异和信息不对称干扰。我们引入SEAM基准，它将四个领域中语义等价的输入配对，这些领域已有标准化的文本和视觉符号。与基于OCR的图文配对不同，SEAM通过在模态间使用不同符号系统，对VLMs的文本-符号和视觉-空间推理能力提供严格的比较评估。在21个当代模型中，我们观察到系统性模态失衡：尽管问题包含语义等价信息，但视觉表现常落后于语言，且跨模态一致性相对较低。错误分析揭示两个主要原因：领域符号中标记化导致的文本感知失败，以及引发幻觉的视觉感知失败。我们还表明结果对视觉变换具有较强鲁棒性。SEAM为衡量和改进模态无关推理建立了受控的、语义等价的设置。
英文摘要：
Evaluating whether vision-language models (VLMs) reason consistently across representations is challenging because modality comparisons are typically confounded by task differences and asymmetric information. We introduce SEAM, a benchmark that pairs semantically equivalent inputs across four domains that have existing standardized textual and visual notations. By employing distinct notation systems across modalities, in contrast to OCR-based image-text pairing, SEAM provides a rigorous comparative assessment of the textual-symbolic and visual-spatial reasoning capabilities of VLMs. Across 21 contemporary models, we observe systematic modality imbalance: vision frequently lags language in overall performance, despite the problems containing semantically equivalent information, and cross-modal agreement is relatively low. Our error analysis reveals two main drivers: textual perception failures from tokenization in domain notation and visual perception failures that induce hallucinations. We also show that our results are largely robust to visual transformations. SEAM establishes a controlled, semantically equivalent setting for measuring and improving modality-agnostic reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
