感知观测站：多模态大语言模型的鲁棒性与基础性表征
The Perceptual Observatory Characterizing Robustness and Grounding in MLLMs
AI总结：
感知观测站通过系统性扰动和真实数据集，评估多模态大语言模型在视觉基础性和关系结构上的鲁棒性，揭示其在扰动下的表现，为模型分析提供系统基础。
摘要：
多模态大语言模型（MLLMs）的最新进展产生了越来越强大的模型，但其感知能力仍缺乏充分表征。在实践中，大多数模型家族在扩展语言组件的同时，重复使用几乎相同的视觉编码器（例如Qwen2.5-VL 3B/7B/72B），这引发了关于进步是反映真正的视觉基础性还是依赖于互联网规模的文本世界知识的重大疑问。现有的评估方法强调端任务准确性，忽视了鲁棒性、归因准确性以及在受控扰动下的推理能力。我们提出了感知观测站，一个框架，用于在诸如：（i）简单的视觉任务，如人脸匹配和文本在视觉中的理解能力；（ii）局部到全局的理解，包括图像匹配、网格指针游戏和属性定位，这些测试通用视觉基础性。每个垂直都使用真实人脸和词语的地面真相数据集，通过像素基增强和扩散基风格化幻觉系统性扰动。感知观测站超越了排行榜准确性，揭示了MLLMs在扰动下如何保持感知基础性和关系结构，为分析当前和未来模型的优势和劣势提供了系统的基础。
英文摘要：
Recent advances in multimodal large language models (MLLMs) have yielded increasingly powerful models, yet their perceptual capacities remain poorly characterized. In practice, most model families scale language component while reusing nearly identical vision encoders (e.g., Qwen2.5-VL 3B/7B/72B), which raises pivotal concerns about whether progress reflects genuine visual grounding or reliance on internet-scale textual world knowledge. Existing evaluation methods emphasize end-task accuracy, overlooking robustness, attribution fidelity, and reasoning under controlled perturbations. We present The Perceptual Observatory, a framework that characterizes MLLMs across verticals like: (i) simple vision tasks, such as face matching and text-in-vision comprehension capabilities; (ii) local-to-global understanding, encompassing image matching, grid pointing game, and attribute localization, which tests general visual grounding. Each vertical is instantiated with ground-truth datasets of faces and words, systematically perturbed through pixel-based augmentations and diffusion-based stylized illusions. The Perceptual Observatory moves beyond leaderboard accuracy to yield insights into how MLLMs preserve perceptual grounding and relational structure under perturbations, providing a principled foundation for analyzing strengths and weaknesses of current and future models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
