参考驱动的野外先验多说话人音频场景生成
Reference-Driven Multi-Speaker Audio Scene Generation from In-the-Wild Priors
AI总结：
提出ScenA方法，利用预训练的文本到音频流匹配基础模型，通过多参考声音和自然语言提示生成多说话人音频场景，并采用高噪声偏置时间步分布解决参考捷径问题，在CoVoMix2-Dialogue基准上优于现有系统。
摘要：
现有的多说话人对话系统通过结构化监督（如每轮标签、多流转录或可学习说话人嵌入）将说话人与话语绑定。这些系统在仅语音的流水线中运行，生成干净的语音序列，缺乏真实对话的环境纹理。我们采取不同的方法。我们的方法ScenA将文本到音频流匹配基础模型（在大规模野外数据上预训练）直接以多个参考声音和描述整个多说话人音频场景的自由形式自然语言提示为条件。利用这样的基础模型使我们能够继承其生成自然、非录音室音频的能力：背景噪声、房间声学、重叠对话和自发的副语言事件，同时添加多说话人控制而无需任何每轮结构。具体地，参考潜在向量被连接到模型的令牌序列中，并通过轻量级的身份感知位置编码进行区分。然而，我们识别出这种方法的一个关键障碍：参考捷径。在标准噪声调度下的训练过程中，模型可以通过声学相似性识别匹配的参考与噪声目标，从而完全绕过文本提示。我们通过高噪声偏置的时间步分布来解决这个问题，迫使模型依赖文本提示进行说话人分配。我们在CoVoMix2-Dialogue基准上评估ScenA，结果表明它在说话人绑定指标上优于现有的多说话人系统，同时生成具有重叠语音、情感发声和环境声音的丰富对话音频。我们的结果证明了使用以自由形式场景描述为条件的通用音频模型，而不是通过仅语音流水线传递结构化对话脚本的优势。
英文摘要：
Existing multi-speaker dialogue systems bind speakers to utterances through structured supervision: per-turn tags, multi-stream transcriptions, or learnable speaker embeddings. These systems operate within speech-only pipelines that produce clean vocal sequences without the ambient texture of real conversations. We take a different approach. Our method, ScenA, conditions a text-to-audio flow-matching foundation model, pretrained on large-scale in-the-wild data, directly on multiple reference voices and a free-form natural language prompt that describes an entire multi-speaker audio scene. Leveraging such a foundational model allows us to inherit its capacity for natural, non-studio audio: background noise, room acoustics, overlapping dialogue, and spontaneous paralinguistic events, while adding multi-speaker control without any per-turn structure. Concretely, reference latents are concatenated into the model's token sequence and distinguished by lightweight identity-aware positional encodings. However, we identify a critical obstacle to this approach: the \textit{Reference Shortcut}. During training under standard noise schedules, the model can identify the matching reference by acoustic similarity to the noisy target, bypassing the text prompt entirely. We address this with a high-noise-biased timestep distribution that forces the model to rely on the text prompt for speaker assignment. We evaluate ScenA on the CoVoMix2-Dialogue benchmark, showing that it outperforms existing multi-speaker systems on speaker-binding metrics while generating rich conversational audio with overlapping speech, emotional vocalizations, and ambient sound. Our results demonstrate the advantage of using a general-purpose audio model conditioned on a free-form scene description, rather than passing structured dialog scripts through a speech-only pipeline.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
