arXiv 2501.061172025-08-13cs.CLcs.AI

Fleurs-SLU：一个面向口语理解的大规模多语言基准

Fleurs-SLU: A Massively Multilingual Benchmark for Spoken Language Understanding

Center For Artificial Intelligence and Data Science, University of Würzburg（人工智能与数据科学中心，乌尔姆大学）；Language Technology Lab, University of Cambridge（语言技术实验室，剑桥大学）；Mila, McGill University and Canada CIFAR AI Chair（Mila，麦吉尔大学及加拿大CIFAR人工智能主席）

Fabian David Schmidt, Ivan Vulić, Goran Glavaš, David Ifeoluwa Adelani

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出多语言口语理解基准Fleurs-SLU，包含102种语言692小时主题分类和92种语言944小时听力问答，评估发现级联系统与闭源语音LLM表现优异，且ASR与SLU性能强相关。

摘要：

口语理解（SLU）对于一半缺乏正式书写系统的现存语言而言是不可或缺的。与高资源语言不同，对于这些语言，我们无法将语音的语义理解卸载给自动语音识别（ASR）与基于文本的大型语言模型（LLMs）的级联系统。即使低资源语言拥有书写系统，由于缺乏双模态语音和文本训练数据，其ASR依然不可靠。然而，多语言SLU的评估仅局限于意图分类或语言识别等浅层任务。为此，我们提出了Fleurs-SLU，一个多语言SLU基准，它包含：涵盖102种语言、用于主题话语分类的692小时语音，以及跨越92种语言、包含944小时语音的基于听力理解的多项选择问答。我们在Fleurs-SLU上广泛评估了端到端语音分类模型、结合语音转文本转录与后续基于LLM分类的级联系统，以及多模态语音LLMs。结果表明，级联系统在多语言SLU中更为鲁棒，尽管预训练良好的语音编码器在主题语音分类中也能表现出竞争力。闭源语音LLMs匹配或超越了级联系统的性能。我们观察到鲁棒的多语言ASR、有效的语音到文本翻译与强大的多语言SLU之间存在强相关性，表明声学与语义语音表示之间存在互惠互利。

英文摘要：

Spoken language understanding (SLU) is indispensable for half of all living languages that lack a formal writing system. Unlike for high-resource languages, for these languages, we cannot offload semantic understanding of speech to the cascade of automatic speech recognition (ASR) and text-based large language models (LLMs). Even if low-resource languages possess a writing system, ASR for these languages remains unreliable due to limited bimodal speech and text training data. Nonetheless, the evaluation of multilingual SLU is limited to shallow tasks such as intent classification or language identification. This is why we present Fleurs-SLU, a multilingual SLU benchmark that encompasses (i) 692 hours of speech for topical utterance classification in 102 languages and (ii) multiple-choice question answering via listening comprehension spanning 944 hours of speech across 92 languages. We extensively evaluate end-to-end speech classification models, cascaded systems that combine speech-to-text transcription with subsequent LLM-based classification, and multimodal speech-LLMs on Fleurs-SLU. Our results show that cascaded systems are more robust in multilingual SLU, though well-pretrained speech encoders can perform competitively in topical speech classification. Closed-source speech-LLMs match or surpass the performance of cascaded systems. We observe a strong correlation between robust multilingual ASR, effective speech-to-text translation, and strong multilingual SLU, indicating mutual benefits between acoustic and semantic speech representations.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。