Audio-Omni: 扩展多模态理解到多功能音频生成与编辑
Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing
AI总结：
Audio-Omni提出首个端到端框架，统一音频生成与编辑，结合多模态理解能力，通过冻结的多模态大语言模型与可训练的扩散变换器实现高保真合成，并构建大规模数据集提升音频编辑性能。
摘要：
近年来多模态模型的进步推动了音频理解、生成和编辑的快速发展。然而，这些能力通常由专门模型处理，真正统一框架的开发仍不充分。尽管一些开创性工作尝试统一音频理解和生成，但往往局限于特定领域。为此，我们引入Audio-Omni，首个端到端框架，统一生成与编辑于通用声音、音乐和语音领域，具备集成的多模态理解能力。我们的架构结合冻结的多模态大语言模型进行高层推理，与可训练的扩散变换器实现高保真合成。为克服音频编辑中的关键数据稀缺问题，我们构建了AudioEdit，一个包含超过一百万对精心编纂编辑对的新大规模数据集。大量实验表明，Audio-Omni在一系列基准测试中达到最先进的性能，优于先前的统一方法，同时在性能上与或优于专门专家模型。除了核心能力外，Audio-Omni还表现出显著的继承能力，包括知识增强的推理生成、上下文生成和零样本跨语言控制，凸显了向通用生成音频智能方向发展的前景。代码、模型和数据集将在https://zeyuet.github.io/Audio-Omni上公开发布。
英文摘要：
Recent progress in multimodal models has spurred rapid advances in audio understanding, generation, and editing. However, these capabilities are typically addressed by specialized models, leaving the development of a truly unified framework that can seamlessly integrate all three tasks underexplored. While some pioneering works have explored unifying audio understanding and generation, they often remain confined to specific domains. To address this, we introduce Audio-Omni, the first end-to-end framework to unify generation and editing across general sound, music, and speech domains, with integrated multi-modal understanding capabilities. Our architecture synergizes a frozen Multimodal Large Language Model for high-level reasoning with a trainable Diffusion Transformer for high-fidelity synthesis. To overcome the critical data scarcity in audio editing, we construct AudioEdit, a new large-scale dataset comprising over one million meticulously curated editing pairs. Extensive experiments demonstrate that Audio-Omni achieves state-of-the-art performance across a suite of benchmarks, outperforming prior unified approaches while achieving performance on par with or superior to specialized expert models. Beyond its core capabilities, Audio-Omni exhibits remarkable inherited capabilities, including knowledge-augmented reasoning generation, in-context generation, and zero-shot cross-lingual control for audio generation, highlighting a promising direction toward universal generative audio intelligence. The code, model, and dataset will be publicly released on https://zeyuet.github.io/Audio-Omni.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
