全模态扩散：基于掩码离散扩散的统一多模态理解与生成
Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion
AI总结：
受离散扩散模型在多领域成功应用启发，提出全模态扩散模型，基于掩码离散扩散模型构建，统一文本、语音和图像的理解与生成，用统一模型直接捕捉离散多模态令牌联合分布，性能优于现有多模态系统。
摘要：
虽然最近的多模态大语言模型取得了显著进展，但主要采用传统自回归架构。近期研究成功将离散扩散模型应用于视觉理解和图像生成等领域。受此启发，我们引入全模态扩散，首个完全基于掩码离散扩散模型构建的任意到任意多模态语言模型，统一跨文本、语音和图像的理解与生成。全模态扩散采用统一的基于掩码的离散扩散模型直接捕获离散多模态令牌上的联合分布。这种方法不仅支持双峰任务，还支持涉及多种模态的更复杂场景。在各种基准测试中，我们的方法优于或与处理两种或更多模态的现有多模态系统相当，突出了扩散模型在推动下一代多模态基础模型方面的巨大潜力。项目网页：this https URL。
英文摘要：
While recent multimodal large language models (MLLMs) have made impressive strides, they predominantly employ a conventional autoregressive architecture as their backbone, leaving significant room to explore effective and efficient alternatives in architectural design. Concurrently, recent studies have successfully applied discrete diffusion models to various domains, such as visual understanding and image generation, revealing their considerable potential as a promising backbone for multimodal systems. Drawing inspiration from these pioneering studies, we introduce Omni-Diffusion, the first any-to-any multimodal language model built entirely on mask-based discrete diffusion models, which unifies understanding and generation across text, speech, and images. Omni-Diffusion employs a unified mask-based discrete diffusion model to directly capture the joint distribution over discrete multimodal tokens. This approach supports not only bimodal tasks but also more complex scenarios involving multiple modalities. On a diverse set of benchmarks, our method outperforms or performs on par with existing multimodal systems that process two or more modalities, highlighting the significant promise of diffusion models in powering the next generation of multimodal foundation models. Project webpage: https://omni-diffusion.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
