URO-Bench：面向端到端语音对话模型的全面评估
URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Models
AI总结：
提出URO-Bench，首个涵盖多语言、多轮对话和副语言学的语音到语音基准，全面评估端到端语音对话模型在理解、推理和口语会话上的能力，揭示现有开源模型在指令遵循和高级音频理解上的不足。
摘要：
大型语言模型（LLMs）的最新进展推动了端到端语音对话模型（SDMs）的显著进步。与基于文本的LLMs不同，SDMs的评估框架应涵盖认知维度（如逻辑推理、知识）和语音相关方面（如副语言线索、音频质量）。然而，目前仍缺乏在语音到语音（S2S）场景下对SDMs的全面评估。为填补这一空白，我们提出了URO-Bench，一个广泛的SDMs基准。值得注意的是，URO-Bench是首个涵盖多语言、多轮对话和副语言学评估的S2S基准。我们的基准分为两个难度级别：基础赛道和专业赛道，各包含20个测试集，评估语音对话模型在理解、推理和口语会话方面的能力。对我们提出的基准的评估显示，当前开源SDMs在日常QA任务中表现相当不错，但在指令遵循能力方面落后于其骨干LLMs，并且也存在灾难性遗忘。它们在副语言信息和音频理解的高级评估中表现仍然欠佳，突出了在这一方向上进一步研究的必要性。我们希望URO-Bench能通过提供对现有模型的多方面评估并帮助跟踪该领域的进展，来促进语音对话模型的发展。
英文摘要：
Recent advances in large language models (LLMs) have driven significant progress in end-to-end spoken dialogue models (SDMs). In contrast to text-based LLMs, the evaluation framework for SDMs should encompass both cognitive dimensions (e.g., logical reasoning, knowledge) and speech-related aspects (e.g., paralinguistic cues, audio quality). However, there is still a lack of comprehensive evaluations for SDMs in speech-to-speech (S2S) scenarios. To address this gap, we propose URO-Bench, an extensive benchmark for SDMs. Notably, URO-Bench is the first S2S benchmark that covers evaluations about multilingualism, multi-round dialogues, and paralinguistics. Our benchmark is divided into two difficulty levels: basic track and pro track, each comprising 20 test sets, evaluating the spoken dialogue model's abilities in Understanding, Reasoning, and Oral conversation. Evaluations on our proposed benchmark reveal that current open-source SDMs perform rather well in daily QA tasks, but lag behind their backbone LLMs in terms of instruction-following ability and also suffer from catastrophic forgetting. Their performance in advanced evaluations of paralinguistic information and audio understanding remains subpar, highlighting the need for further research in this direction. We hope that URO-Bench can facilitate the development of spoken dialogue models by providing a multifaceted evaluation of existing models and helping to track progress in this area.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
