EXAM²：扩展多语言与多模态分析中的音频理解能力

中文摘要 AI 辅助

近期的大型音频语言模型（LALMs）在音频理解领域取得了显著进展，但现有评估大多局限于英语和狭窄的音频领域，此前的基准通常仅关注单一音频模态（即语音、声音或音乐），限制了对这些模型在不同视觉场景下泛化能力的系统研究。本文提出EXAM²，这是一个覆盖6种语言及多种模态的多语言多模态音频理解基准，涵盖语音、声音、音乐、混合音频场景及视觉图像。EXAM²将视觉信息与异构音频输入相结合，可更真实地评估场景感知音频推理与跨模态理解能力，包含5667个多项选择题、22614个图像实例及135684个多语言翻译。我们评估了当前最先进的开源、专有LALMs及多模态大语言模型（LLMs），发现其在多语言和跨模态理解方面存在显著性能差距；此外，我们提出在EXAM²训练集上微调的轻量级融合模型Gemma3n-EXAM²，在多语言场景下较强基线实现了最高12.4%的提升，在多模态评估中实现了21.7%的增益。实验结果表明EXAM²是一个具有挑战性的基准，将推动未来多语言多模态音频智能研究的发展。

英文摘要

Recent large audio language models (LALMs) have achieved impressive progress in audio understanding. However, existing evaluations remain largely constrained to English and narrow audio domains. Prior benchmarks typically focus on a single audio modality, i.e., speech, sound, or music, limiting the systematic investigation into how these models generalize across diverse visual scenarios. In this paper, we introduce EXAM$^2$, a benchmark for multilingual and multimodal audio understanding spanning six languages and multiple modalities, including speech, sound, music, mixed-audio settings, and visual images. By incorporating visual information alongside heterogeneous audio inputs, EXAM$^2$ enables more realistic evaluation of scene-aware audio reasoning and cross-modal comprehension. EXAM$^2$ comprises $5,667$ multiple-choice questions, $22,614$ image instances, and $135,684$ multilingual translations. We evaluate state-of-the-art open-source and proprietary LALMs as well as multimodal LLMs, revealing substantial performance gaps in multilingual and cross-modal understanding. Furthermore, we propose Gemma3n-EXAM$^2$, a lightweight fusion-model fine-tuned on EXAM$^2$-train, achieves up to $15.8\%$ improvement in multilingual settings and $16.5\%$ gains in multimodal evaluation over a strong baseline. Empirical results establish EXAM$^2$ as a challenging benchmark and pioneer future multilingual and multimodal audio intelligence research.

补充信息

备注

9 pages, 2 figures