arXiv 2508.172902025-08-24cs.AIcs.LG

MEENA（PersianMMMU）：用于N级评估的多模态多语言教育考试

MEENA (PersianMMMU): Multimodal-Multilingual Educational Exams for N-level Assessment

Computer Engineering Department, Sharif University of Technology, Iran（谢尔盖大学计算机工程系，伊朗）；Qatar Computing Research Institute, Qatar（卡塔尔计算研究所，卡塔尔）；Computer Engineering Department, University of Isfahan, Iran（伊斯法罕大学计算机工程系，伊朗）；Independent Researcher（独立研究者）

Omid Ghahroodi, Arshia Hemmat, Marzia Nouri, Seyed Mohammad Hadi Hosseini, Doratossadat Dastgheib, Mohammad Vali Sanian, Alireza Sahebi, Reihaneh Zohrabi, Moham… 展开作者

Omid Ghahroodi, Arshia Hemmat, Marzia Nouri, Seyed Mohammad Hadi Hosseini, Doratossadat Dastgheib, Mohammad Vali Sanian, Alireza Sahebi, Reihaneh Zohrabi, Mohammad Hossein Rohban, Ehsaneddin Asgari, Mahdieh Soleymani Baghshah

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对大型视觉语言模型（VLMs）多聚焦英语的局限，本文引入首个波斯语多模态多语言教育考试数据集MEENA（PersianMMMU），含7500个波斯语和3000个英语问题，覆盖多学科与教育水平，助力评估波斯语VLMs能力并推动其发展。

摘要：

大型视觉语言模型（VLMs）的最新进展主要集中在英语上，对其他语言的关注有限。为解决这一差距，我们引入MEENA（又称PersianMMMU），这是首个旨在评估波斯语VLMs在科学、推理和人类水平理解任务上表现的数据集。该数据集包含约7500个波斯语问题和3000个英语问题，涵盖推理、数学、物理、图表、图表以及波斯艺术和文学等广泛主题。MEENA的主要特点包括：（1）覆盖从小学到高中等不同教育水平的多样化学科；（2）包含难度等级和描述性答案的丰富元数据；（3）保留文化细微差别的原始波斯语数据；（4）用于评估跨语言性能的双语结构；（5）一系列评估各种能力的多样化实验，包括整体性能、模型对图像的注意力能力及其产生幻觉的倾向。我们希望该基准有助于提升VLMs在英语之外的能力。

英文摘要：

Recent advancements in large vision-language models (VLMs) have primarily focused on English, with limited attention given to other languages. To address this gap, we introduce MEENA (also known as PersianMMMU), the first dataset designed to evaluate Persian VLMs across scientific, reasoning, and human-level understanding tasks. Our dataset comprises approximately 7,500 Persian and 3,000 English questions, covering a wide range of topics such as reasoning, mathematics, physics, diagrams, charts, and Persian art and literature. Key features of MEENA include: (1) diverse subject coverage spanning various educational levels, from primary to upper secondary school, (2) rich metadata, including difficulty levels and descriptive answers, (3) original Persian data that preserves cultural nuances, (4) a bilingual structure to assess cross-linguistic performance, and (5) a series of diverse experiments assessing various capabilities, including overall performance, the model's ability to attend to images, and its tendency to generate hallucinations. We hope this benchmark contributes to enhancing VLM capabilities beyond English.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。