Sci-Rho：面向STEM问题的多语言视觉基础符号基准
Sci-Rho: A Multilingual Visually-Grounded Symbolic Benchmark for STEM Problems
AI总结：
提出Sci-Rho，一个多语言、视觉基础的STEM问题动态基准，包含4242个模板和42420个实例，评估17个VLM发现最差精度与平均精度存在差距，且小模型跨语言性能下降。
摘要：
符号基准已成为评估模型在STEM相关问题微小修改下鲁棒性的关键方法。然而，现有符号基准大多局限于数学推理，缺乏视觉基础，且主要以英语为主。在这项工作中，我们引入了Sci-Rho（科学鲁棒性），一个面向视觉基础STEM问题的动态基准，涵盖五个学科和七种语言，包含由领域专家（包括奥林匹克奖牌得主）精心设计的4,242个问题模板（每种语言606个）。每个模板实现为可执行的Python代码，通过改变数值、视觉模式、几何形状、颜色方案和函数类型，生成多样但等价的问题实例，总共产生42,420个实例，每个实例都配有推理步骤和真实解决方案。我们评估了17个最先进的VLM，发现最差情况准确率（定义为模型在每种生成变体上均正确回答的问题模板比例）与平均准确率之间存在明显差距。我们还发现，较小的模型在不同语言上表现出显著的性能下降，而专有模型和较大模型保持鲁棒。步骤级评估反映了相同的趋势，揭示了平均F1与最差情况F1分数之间的显著差距。最后，我们对VLM注意力头的检查显示，图像标记与文本标记的相对注意力分配存在显著的跨语言变化。我们的工作强调了超越静态基准的评估作为衡量VLM质量指标的重要性。
英文摘要：
Symbolic benchmarks have emerged as a key approach to assess model robustness under minor modifications to STEM-related questions. However, existing symbolic benchmarks mostly remain limited to mathematical reasoning, lack visual grounding, and are predominantly in English. In this work, we introduce Sci-Rho (Science Rhobustness), a dynamic benchmark for visually-grounded STEM problems spanning five subjects and seven languages, comprising 4,242 problem templates (606 per language) crafted by domain experts, including Olympiad medalists. Each template is implemented as executable Python code that generates diverse but equivalent problem instances by varying numerical values, visual patterns, geometric shapes, color schemes, and function types, resulting in 42,420 instances in total, each paired with reasoning steps and ground-truth solutions. We evaluated 17 state-of-the-art VLMs and discovered a noticeable gap between worst-case accuracy (defined as the proportion of problem templates that a model answers correctly across every generated variation) and average accuracy. We also discovered that smaller models show noticeable performance degradation across languages, whereas proprietary and larger models remain robust. Step-level evaluation reflects this same trend, revealing a significant gap between average F1 and worst-case F1 scores. Finally, our inspection of attention heads of a VLM reveals substantial cross-lingual variation in the relative attention allocated to image tokens compared to text tokens. Our work highlights the importance of evaluation beyond static benchmarks as a metric to measure the quality of VLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
