并非真正的多语言：脚本一致性作为VLM评估中缺失的维度
Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation
AI总结：
提出PuMVR基准，评估10个VLM在旁遮普语三种文字上的表现，发现显著的脚本差距，并提出脚本一致性率（SCR）作为必要评估指标。
摘要：
当前视觉语言模型（VLM）的多语言评估假设语言与正字法一一对应，忽略了使用多种文字语言的数十亿用户。我们引入了PuMVR（旁遮普多模态视觉推理），这是一个包含1000个严格平行图像-文本实例的基准，覆盖旁遮普语的三种活跃文字：古木基文、沙穆基文和罗马文。评估10个最先进的VLM，我们暴露了一个显著且系统的脚本差距。模型经常在一种文字上解决视觉任务，而在另一种文字上失败，准确率差异高达16%。关键的是，视觉输入均匀地提升了绝对性能，但并未缩小正字法差距。此外，跨文字的上下文迁移非常脆弱，揭示了脚本锁定的知识表示。通过所有文字对的McNemar检验支持，我们的发现表明当前的“多语言”VLM并非真正的多文字。我们提出脚本一致性率（SCR），在我们的基准上低至24.8%，作为脚本无关评估的强制性指标，以确保公平的AI访问。数据和代码可在以下网址获取：this https URL。
英文摘要：
Current multilingual evaluations for Vision-Language Models (VLMs) assume a one-to-one mapping between language and orthography, overlooking billions of users of multi-script languages. We introduce PuMVR (Punjabi Multimodal Visual Reasoning), a benchmark of 1,000 strictly parallel image-text instances across Punjabi's three active scripts: Gurmukhi, Shahmukhi, and Roman. Evaluating 10 state-of-the-art VLMs, we expose a substantial and systematic Script Gap. Models frequently solve visual tasks in one script while failing identical tasks in another, with accuracy deltas reaching 16%. Crucially, visual input boosts absolute performance uniformly yet does not close the orthographic gap. Furthermore, cross-script in-context transfer is highly brittle, exposing script-locked knowledge representation. Supported by McNemar tests across all script pairs, our findings demonstrate that current "multilingual" VLMs are not truly multi-script. We propose the Script Consistency Rate (SCR), which falls as low as 24.8% on our benchmark, as a mandatory metric for script-agnostic evaluation to ensure equitable AI access. Data and code are available at: https://github.com/prabhjotschugh/Not-Truly-Multilingual-PuMVR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
