跨学术与文化领域评估乌克兰语理解能力的多模态模型基准测试
Benchmarking Multimodal Models for Ukrainian Language Understanding Across Academic and Cultural Domains
AI总结：
本文构建了基于乌克兰大学入学考试的4300余题多模态基准ZNO-Vision，评估多模型在12个学科上的表现，并首次开展乌克兰语多模态文本生成评估及文化知识测试，以推动低资源语言多模态能力发展。
摘要：
虽然以英语为中心的多模态模型评估是一个活跃的研究领域，拥有众多基准测试，但对于低资源和中等资源语言，基准测试或评估套件严重缺乏。我们引入了ZNO-Vision，这是一个源自标准化大学入学考试（ZNO）的综合性乌克兰语多模态基准测试。该基准测试包含超过4300个由专家精心设计的问题，涵盖12个学术学科，包括数学、物理、化学和人文学科。我们评估了开源模型和API提供商的表现，发现只有少数模型的表现高于基线。除了新的基准测试，我们还进行了乌克兰语多模态文本生成的首次评估研究：我们在Multi30K-UK数据集上测量了图像描述生成质量，将VQA基准测试翻译成乌克兰语，并测量了相对于原始英文版本的性能下降。最后，我们从文化视角测试了几个模型对民族美食知识的掌握情况。我们相信，我们的工作将推动乌克兰语多模态生成能力的发展，并且我们的方法可能对其他低资源语言有用。
英文摘要：
While the evaluation of multimodal English-centric models is an active area of research with numerous benchmarks, there is a profound lack of benchmarks or evaluation suites for low- and mid-resource languages. We introduce ZNO-Vision, a comprehensive multimodal Ukrainian-centric benchmark derived from standardized university entrance examination (ZNO). The benchmark consists of over 4,300 expert-crafted questions spanning 12 academic disciplines, including mathematics, physics, chemistry, and humanities. We evaluated the performance of both open-source models and API providers, finding that only a handful of models performed above baseline. Alongside the new benchmark, we performed the first evaluation study of multimodal text generation for the Ukrainian language: we measured caption generation quality on the Multi30K-UK dataset, translated the VQA benchmark into Ukrainian, and measured performance degradation relative to original English versions. Lastly, we tested a few models from a cultural perspective on knowledge of national cuisine. We believe our work will advance multimodal generation capabilities for the Ukrainian language and our approach could be useful for other low-resource languages.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
