自由文本提示的统一语音与声音合成
Unified Synthesis of Compositional Speech and Sound from Free-Form Text Prompts
AI总结：
提出PlanAudio框架，利用大语言模型推理能力和语义潜在思维链机制，直接从自由文本生成包含语音和声音的统一音频。
摘要：
音频生成已取得显著进展，但合成语音与声音自然组合的统一音频仍具挑战。当前方法要么依赖分离的流水线，无法捕捉细粒度交互，要么需要结构化输入和外部文本重写，限制了自由文本提示的灵活性。本文提出新任务：自由文本提示到统一音频生成，旨在直接从无约束自然语言合成包含语音、声音及其复合的统一音频。为此，我们提出PlanAudio，一个统一的、基于自回归LLM的框架。首先，它利用LLM内在推理能力简化模型架构，而非传统文本编码器。其次，引入语义潜在思维链机制，一种隐式规划机制，连接高层语义理解与低层声学合成。此外，我们创建PlanAudio-Bench，一个专门评估复合音频场景的基准。我们在语音、声音及其复合场景下进行评估。结果表明，PlanAudio普遍优于现有流水线和统一基线，同时与专为单一场景设计的模型保持竞争力。进一步分析揭示了语义潜在CoT相对于其他CoT机制的优越性，并强调了连续多场景训练课程的重要性。
英文摘要：
Audio generation has made significant progress, yet synthesizing unified audio where speech and sounds are naturally composited remains a challenge. Current methods either rely on disjoint pipelines, which fail to capture fine-grained interactions, or require structured inputs and external text rewriting, which limits the flexibility of free-form text prompts. In this paper, we introduce a new task: Free-Form-Text-Prompt-to-Unified-Audio generation, which aims to directly synthesize unified audio containing speech, sound, and their composites from unconstrained natural language. To address this task, we propose PlanAudio, a unified, autoregressive LLM-based framework. First, it simplifies the model architecture by leveraging intrinsic LLM reasoning capability instead of traditional text encoders. Second, it introduces a semantic latent chain-of-thought mechanism, an implicit planning mechanism that bridges high-level semantic understanding and low-level acoustic synthesis. Furthermore, we create PlanAudio-Bench, a specialized benchmark for evaluating composite audio scenarios. We perform evaluations in the scenarios of speech, sound, and their composites. The results demonstrate that PlanAudio generally outperforms the existing pipeline and unified baselines, while staying competitive with models designed for a single scenario. Our analysis further reveals the superiority of semantic latent CoT over other CoT mechanisms and highlights the importance of continuous multi-scenario training curricula.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
