FD-Bench：为全双工口语对话系统设计的全双工基准测试流水线
FD-Bench: A Full-Duplex Benchmarking Pipeline Designed for Full Duplex Spoken Dialogue Systems
AI总结：
针对全双工口语对话系统缺乏打断等场景评测的问题，本文提出融合LLM、TTS和ASR的FD-Bench流水线，并对三个开源模型进行40余小时语音实验，揭示其在频繁打断与噪声下的不足。
摘要：
全双工口语对话系统（FDSDS）允许实时用户打断和反馈通道，与依赖轮流发言的传统SDS相比，能够实现更自然的人机交互。然而，现有基准缺乏面向FD场景的指标，例如评估模型在用户打断期间的性能。本文中，我们提出一个利用LLM、TTS和ASR的综合FD基准测试流水线，以弥补这一空白。该流水线通过多种新颖指标，评估FDSDS处理用户打断、管理延迟以及在挑战性场景中保持鲁棒性的能力。我们使用超过40小时的生成语音，将该基准应用于三个开源FDSDS（Moshi、Freeze-omni和VITA-1.5），包含293段模拟对话和1,200次打断。结果表明，在频繁打断和噪声条件下，所有模型仍面临挑战，例如未能对用户打断作出响应。演示、数据和代码将会发布。
英文摘要：
Full-duplex spoken dialogue systems (FDSDS) enable more natural human-machine interactions by allowing real-time user interruptions and backchanneling, compared to traditional SDS that rely on turn-taking. However, existing benchmarks lack metrics for FD scenes, e.g., evaluating model performance during user interruptions. In this paper, we present a comprehensive FD benchmarking pipeline utilizing LLMs, TTS, and ASR to address this gap. It assesses FDSDS's ability to handle user interruptions, manage delays, and maintain robustness in challenging scenarios with diverse novel metrics. We applied our benchmark to three open-source FDSDS (Moshi, Freeze-omni, and VITA-1.5) using over 40 hours of generated speech, with 293 simulated conversations and 1,200 interruptions. The results show that all models continue to face challenges, such as failing to respond to user interruptions, under frequent disruptions and noisy conditions. Demonstrations, data, and code will be released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
