mSTEB：面向语音与文本任务的大规模多语言大语言模型评测
mSTEB: Massively Multilingual Evaluation of LLMs on Speech and Text Tasks
AI总结：
本文提出多语言多模态评测基准 mSTEB，覆盖语音与文本上的语言识别、分类、问答和翻译任务，通过评测主流闭源与开源大语言模型，揭示高资源与低资源语言间的显著性能差距，呼吁加大对低资源语言覆盖的投入。
摘要：
大语言模型（LLMs）在广泛任务上展现了出色性能，包括语音等多模态场景。然而，其评测往往仅限于英语和少数高资源语言。对于低资源语言，目前缺乏标准化的评测基准。本文通过引入 mSTEB 这一新基准来填补这一空白，该基准用于评测大语言模型在语音和文本两种模态上的语言识别、文本分类、问答和翻译等广泛任务中的表现。我们评测了 Gemini 2.0 Flash 和 GPT-4o (Audio) 等领先大语言模型，以及 Qwen 2 Audio 和 Gemma 3 27B 等最先进开源模型的性能。我们的评测显示，高资源语言与低资源语言之间存在巨大性能差距，尤其是非洲和美洲/大洋洲地区使用的语言。我们的研究结果表明，需要加大投入以解决这些语言在大语言模型覆盖中代表性不足的问题。
英文摘要：
Large Language models (LLMs) have demonstrated impressive performance on a wide range of tasks, including in multimodal settings such as speech. However, their evaluation is often limited to English and a few high-resource languages. For low-resource languages, there is no standardized evaluation benchmark. In this paper, we address this gap by introducing mSTEB, a new benchmark to evaluate the performance of LLMs on a wide range of tasks covering language identification, text classification, question answering, and translation tasks on both speech and text modalities. We evaluated the performance of leading LLMs such as Gemini 2.0 Flash and GPT-4o (Audio) and state-of-the-art open models such as Qwen 2 Audio and Gemma 3 27B. Our evaluation shows a wide gap in performance between high-resource and low-resource languages, especially for languages spoken in Africa and Americas/Oceania. Our findings show that more investment is needed to address their under-representation in LLMs coverage.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
