ZipVoice-Dialog: 基于流匹配的非自回归语音对话生成
ZipVoice-Dialog: Non-Autoregressive Spoken Dialogue Generation with Flow Matching
AI总结：
本文提出ZipVoice-Dialog模型，通过流匹配和课程学习策略提升语音对话生成的推理速度和准确性，同时发布OpenDialog数据集以促进研究发展。
摘要：
生成语音对话本质上比单人语音合成更复杂，因为它需要真实的轮询和保持不同的说话人音色。尽管现有自回归（AR）模型有所进展，但它们通常存在高推理延迟和稳定性问题。为克服这些限制，我们提出了ZipVoice-Dialog，一种基于流匹配的非自回归（NAR）零样本语音对话生成模型。观察到将 vanilla 流匹配应用于对话生成会导致语音可懂度和轮询精度差，我们引入了两种简单但有效的适应流匹配架构的方法：（1）课程学习策略以确保语音-文本对齐，（2）说话人-轮次嵌入以精确控制说话人轮询。此外，我们引入了专门的策略来支持立体对话生成。认识到该领域缺乏训练数据集，我们整理并发布了OpenDialog，第一个大规模（6.8k小时）开源语音对话数据集，来源于真实场景语音数据。此外，为了公平和严谨的评估，我们建立了基准以全面评估对话生成模型。实验表明所提出方法和数据集的有效性，显示ZipVoice-Dialog在推理速度、可懂度、说话人轮询准确性和说话人相似性方面均表现优异。我们的代码、模型检查点和OpenDialog数据集可在https://github.com/k2-fsa/ZipVoice公开获取。
英文摘要：
Generating spoken dialogue is inherently more complex than monologue text-to-speech (TTS), as it demands both realistic turn-taking and the maintenance of distinct speaker timbres. While existing autoregressive (AR) models have made progress, they often suffer from high inference latency and stability issues. To overcome these limitations, we propose ZipVoice-Dialog, a non-autoregressive (NAR) zero-shot spoken dialogue generation model based on flow-matching. Observing that applying vanilla flow-matching to dialogue generation leads to poor speech intelligibility and turn-taking precision, we introduce two simple yet effective methods to adapt flow-matching architectures for dialogue generation: (1) a curriculum learning strategy to ensure robust speech-text alignment, and (2) speaker-turn embeddings to govern precise speaker turn-taking. Additionally, we introduce dedicated strategies to support stereo dialogue generation. Recognizing the lack of training datasets in this field, we curate and release OpenDialog, the first large-scale (6.8k hours) open-source spoken dialogue dataset derived from in-the-wild speech data. Moreover, for fair and rigorous evaluations, we established a benchmark to comprehensively evaluate dialogue generation models. Experiments demonstrate the effectiveness of the proposed methods and dataset, showing that ZipVoice-Dialog achieves superior performance in inference speed, intelligibility, speaker turn-taking accuracy, and speaker similarity. Our code, model checkpoints, and the OpenDialog dataset are publicly available at https://github.com/k2-fsa/ZipVoice.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
