CSVQA：用于评估视觉语言模型STEM推理能力的中文多模态基准
CSVQA: A Chinese Multimodal Benchmark for Evaluating STEM Reasoning Capabilities of VLMs
AI总结：
针对现有多模态基准缺乏真实科学推理场景的问题，研究提出中文多模态基准CSVQA，包含1378道STEM领域视觉问答题目及配套评估协议，评估15款VLM后发现最优模型准确率仅49.6%，凸显提升该类模型科学推理能力的紧迫性。
摘要：
视觉语言模型（VLM）已在多模态理解领域取得显著进展，但其科学推理能力仍未得到充分评估。当前的多模态基准主要评估通用图像理解或文本驱动的推理，缺乏需要将领域特定知识与视觉证据分析相结合的真实科学场景。为填补这一空白，我们提出CSVQA，这是一个专门设计用于通过领域相关视觉问答评估科学推理能力的诊断性多模态基准。我们的基准包含1378个精心构建的问答对，覆盖多个STEM学科，每个问答对都需要领域知识、视觉证据整合和高阶推理能力。与此前的多模态基准相比，CSVQA更强调真实世界科学内容和复杂推理。我们还提出了一套严格的评估协议，可基于精心整理的解释，系统评估模型预测是否有有效的中间推理步骤支撑。我们在该基准上对15个VLM开展了综合评估，结果显示模型间存在显著性能差距，即使是排名第一的专有模型也仅达到49.6%的准确率。这一实证结果凸显了提升VLM科学推理能力的迫切需求。我们的CSVQA已在https://huggingface.co/datasets/Skywork/CSVQA发布。
英文摘要：
Vision-Language Models (VLMs) have demonstrated remarkable progress in multimodal understanding, yet their capabilities for scientific reasoning remain inadequately assessed. Current multimodal benchmarks predominantly evaluate generic image comprehension or text-driven reasoning, lacking authentic scientific contexts that require domain-specific knowledge integration with visual evidence analysis. To fill this gap, we present CSVQA, a diagnostic multimodal benchmark specifically designed for evaluating scientific reasoning through domain-grounded visual question answering. Our benchmark features 1,378 carefully constructed question-answer pairs spanning diverse STEM disciplines, each demanding domain knowledge, integration of visual evidence, and higher-order reasoning. Compared to prior multimodal benchmarks, CSVQA places greater emphasis on real-world scientific content and complex reasoning. We additionally propose a rigorous evaluation protocol to systematically assess whether model predictions are substantiated by valid intermediate reasoning steps based on curated explanations. Our comprehensive evaluation of 15 VLMs on this benchmark reveals notable performance disparities, as even the top-ranked proprietary model attains only 49.6% accuracy. This empirical evidence underscores the pressing need for advancing scientific reasoning capabilities in VLMs. Our CSVQA is released at https://huggingface.co/datasets/Skywork/CSVQA
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
