M3Kang: 评估视觉-语言模型在多语言多模态数学推理中的表现
M3Kang: Evaluating Multilingual Multimodal Mathematical Reasoning in Vision-Language Models
AI总结：
M3Kang是一个大规模多语言多模态数学推理数据集，用于评估视觉-语言模型在多语言数学推理中的表现，通过基准测试揭示模型在基础数学和图表推理上的不足，并展示多语言技术在多模态设置中的有效性。
摘要：
尽管最先进的视觉-语言模型（VLMs）已显示出强大的推理能力，但它们在多语言数学推理方面的表现仍处于探索阶段，尤其是在与人类表现相比时。为弥合这一差距，我们引入了M3Kang，这是首个大规模多语言、多模态数学推理数据集，用于VLMs。它源自世界最大的数学竞赛——Kangaroo Math Competition，该竞赛每年吸引来自超过90个国家的年龄不足18岁的超过600万名参与者。M3Kang包含1,747个独特的多项选择问题，按年级难度组织，翻译成108种文化多样性的语言，其中一些语言包含用于解决问题的图表。使用此数据集，我们对闭源和开源的最新模型进行了广泛的基准测试。我们发现，尽管有最近的进展，模型在基本数学和基于图表的推理上仍存在困难，性能随着语言存在和模型大小而扩大，但不随年级水平扩大。我们还发现，多语言技术可以有效地扩展到多模态设置，从而在基线方法上取得显著改进。我们的分析还纳入了超过68,000名学生的性能数据，使可以直接与人类表现进行比较。我们开源了M3Kang，包括仅英文子集M2Kang，以及用于构建数据集的框架和代码库。
英文摘要：
Despite state-of-the-art vision-language models (VLMs) have demonstrated strong reasoning capabilities, their performance in multilingual mathematical reasoning remains underexplored, particularly when compared to human performance. To bridge this gap, we introduce M3Kang, the first massively multilingual, multimodal mathematical reasoning dataset for VLMs. It is derived from the Kangaroo Math Competition, the world's largest mathematics contest, which annually engages over six million participants under the age of 18 across more than 90 countries. M3Kang includes 1,747 unique multiple-choice problems organized by grade-level difficulty, with translations into 108 culturally diverse languages, some of them including diagrams essential for solving them. Using this dataset, we conduct extensive benchmarking on both closed- and open-source SOTA models. We observe that, despite recent advances, models still struggle with basic math and diagram-based reasoning, with performance scaling with language presence and model size, but not with grade level. We also find that multilingual techniques can be effectively extended to the multimodal setting, resulting in significant improvements over baseline approaches. Our analysis also incorporates performance data from over 68,000 students, enabling direct comparison with human performance. We are open-sourcing M3Kang, including the English-only subset M2Kang, along with the framework and codebase used to construct the dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
