WavChat：口语对话模型综述
WavChat: A Survey of Spoken Dialogue Models
AI总结：
针对口语对话系统缺乏全面综述的问题，本文按时间顺序梳理现有系统并分为级联与端到端范式，深入分析核心技术、数据集及评估体系，为该领域学术研究与工业应用提供参考。
摘要：
以GPT-4o等系统为代表的口语对话模型近期进展在语音领域引起广泛关注。与传统由语音识别（ASR）、大型语言模型（LLMs）和文本转语音（TTS）组成的三层级联口语对话模型相比，现代口语对话模型具备更强的智能性。这些先进模型不仅能理解音频、音乐及其他语音相关特征，还能捕捉语音中的风格和音色特点；此外，它们能生成高质量、低延迟的多轮语音响应，通过边听边说能力实现实时交互。尽管口语对话系统取得了进展，但缺乏系统梳理和分析这些系统及底层技术的全面综述。为此，我们首先按时间顺序整理了现有口语对话系统，并将其分为级联和端到端两种范式；接着深入概述了口语对话模型的核心技术，涵盖语音表示、训练范式、流式处理、双工通信和交互能力等方面，每个部分均讨论了技术局限性并提出未来研究方向；此外，我们从训练和评估角度全面回顾了相关数据集、评估指标和基准。我们希望本综述能推动口语对话系统领域的学术研究和工业应用发展，相关资料可在https://github.com/jishengpeng/WavChat获取。
英文摘要：
Recent advancements in spoken dialogue models, exemplified by systems like GPT-4o, have captured significant attention in the speech domain. Compared to traditional three-tier cascaded spoken dialogue models that comprise speech recognition (ASR), large language models (LLMs), and text-to-speech (TTS), modern spoken dialogue models exhibit greater intelligence. These advanced spoken dialogue models not only comprehend audio, music, and other speech-related features, but also capture stylistic and timbral characteristics in speech. Moreover, they generate high-quality, multi-turn speech responses with low latency, enabling real-time interaction through simultaneous listening and speaking capability. Despite the progress in spoken dialogue systems, there is a lack of comprehensive surveys that systematically organize and analyze these systems and the underlying technologies. To address this, we have first compiled existing spoken dialogue systems in the chronological order and categorized them into the cascaded and end-to-end paradigms. We then provide an in-depth overview of the core technologies in spoken dialogue models, covering aspects such as speech representation, training paradigm, streaming, duplex, and interaction capabilities. Each section discusses the limitations of these technologies and outlines considerations for future research. Additionally, we present a thorough review of relevant datasets, evaluation metrics, and benchmarks from the perspectives of training and evaluating spoken dialogue systems. We hope this survey will contribute to advancing both academic research and industrial applications in the field of spoken dialogue systems. The related material is available at https://github.com/jishengpeng/WavChat.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
