UAT: 统一音频-文本扩散用于音频生成、编辑和字幕生成
UAT: Unified Audio-Text Diffusion for Audio Generation, Editing, and Captioning
AI总结：
提出UAT框架，通过连续潜扩散和掩码离散扩散实现音频生成、编辑和字幕生成的统一，在保持高保真音频合成的同时取得有竞争力的字幕性能。
摘要：
音频生成和音频到文本理解在很大程度上仍然是分离的，扩散模型主导高保真合成，而自回归（AR）语言模型驱动字幕生成和语义预测。现有的统一方法通常依赖于异构模块或AR中心建模，这可能阻碍联合优化并限制声学保真度。我们提出UAT，据我们所知，这是第一个支持统一音频生成、编辑和字幕生成的扩散中心框架。UAT将音频的连续潜扩散与文本的掩码离散扩散相结合，在共享的双流骨干中实现双向音频-文本建模。实验表明，UAT在保持强大的音频生成和编辑能力的同时，实现了有竞争力的字幕性能，展示了声学合成和语义预测之间的良好平衡。演示样本可在 https://UAT-demo.github.io 获取。
英文摘要：
Audio generation and audio-to-text understanding remain largely separate, with diffusion models dominating high-fidelity synthesis and autoregressive (AR) language models driving captioning and semantic prediction. Existing unified approaches typically rely on either heterogeneous modules or AR-centric modeling, which can hinder joint optimization and limit acoustic fidelity. We present UAT, to our knowledge, the first diffusion-centric framework that supports unified audio generation, editing, and captioning. UAT couples continuous latent diffusion for audio with masked discrete diffusion for text, enabling bidirectional audio-text modeling within a shared dual-stream backbone. Experiments show that UAT preserves strong audio generation and editing capabilities while achieving competitive captioning performance, demonstrating a favorable balance between acoustic synthesis and semantic prediction. Demo samples are available at https://UAT-demo.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
