KoALa-Bench：评估大型音频语言模型在韩语语音理解与忠实性上的表现
KoALa-Bench: Evaluating Large Audio Language Models on Korean Speech Understanding and Faithfulness
AI总结：
本文提出KoALa-Bench，一个用于评估大型音频语言模型在韩语语音理解和忠实性方面的基准，包含六个任务，涵盖语音识别、翻译、问答和指令跟随等核心能力，并包含韩国特定知识内容。
摘要：
近年来，大型音频语言模型（LALMs）的进步使多语言语音理解成为可能。然而，针对非英语语言的LALM评估基准仍然稀缺，韩语便是其中之一。本文介绍KoALa-Bench，一个全面评估LALM韩语语音理解和语音忠实性的基准。特别是，KoALa-Bench包含六个任务。四个任务评估基本语音理解能力，包括自动语音识别、语音翻译、语音问答和语音指令跟随，而剩余两个任务评估语音忠实性，受我们观察到的某些LALM未能充分利用语音模态的启发。此外，为了反映韩国特定知识，我们的基准包含韩国大学入学考试的听力问题以及涵盖韩国文化领域的相关内容。我们对六个模型进行了广泛的实验，包括白盒和黑盒模型。我们的基准、评估代码和排行榜已公开在https://ksbench.github.io/Korean-Benchmark/。
英文摘要：
Recent advances in large audio language models (LALMs) have enabled multilingual speech understanding. However, benchmarks for evaluating LALMs remain scarce for non-English languages, with Korean being one such underexplored case. In this paper, we introduce KoALa-Bench, a comprehensive benchmark for evaluating Korean speech understanding and speech faithfulness of LALMs. In particular, KoALa-Bench comprises six tasks. Four tasks evaluate fundamental speech understanding capabilities, including automatic speech recognition, speech translation, speech question answering, and speech instruction following, while the remaining two tasks evaluate speech faithfulness, motivated by our observation that several LALMs often fail to fully leverage the speech modality. Furthermore, to reflect Korea-specific knowledge, our benchmark incorporates listening questions from the Korean college scholastic ability test as well as content covering Korean cultural domains. We conduct extensive experiments across six models, including both white-box and black-box ones. Our benchmark, evaluation code, and leaderboard are publicly available at https://ksbench.github.io/Korean-Benchmark/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
