arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.04056cs.CL
收藏

弥合离散-连续鸿沟:通过耦合流形离散吸收扩散实现统一多模态生成

Bridging the Discrete-Continuous Gap: Unified Multimodal Generation via Coupled Manifold Discrete Absorbing Diffusion

Sun Yat-sen University(中山大学);Guangdong Key Lab of Big Data Analysis & Processing(广东省大数据分析与处理重点实验室);X-Era AI Lab(X-Era AI实验室)

Yuanfeng Xu, Yuhao Chen, Liang Lin, Guangrun Wang

AI总结:

CoM-DAD通过耦合流形离散吸收扩散框架,实现离散与连续数据的统一多模态生成,解决多模态对齐与训练稳定性问题。

摘要:

生成模型的分裂成用于离散数据(文本)的自回归方法和用于连续数据(图像)的扩散方法,阻碍了真正统一的多模态系统的发展。尽管掩码语言模型(MLMs)提供了高效的双向上下文,但它们通常缺乏自回归模型的生成保真度和扩散模型的语义连续性。此外,将掩码生成扩展到多模态设置会引入严重的对齐挑战和训练不稳定性。在本工作中,我们提出CoM-DAD(耦合流形离散吸收扩散),一种新的概率框架,将多模态生成重新表述为分层双过程。CoM-DAD将高级语义规划与低级令牌合成解耦。首先,我们通过连续潜在扩散过程建模语义流形;其次,我们将令牌生成视为离散吸收扩散过程,受变量速率噪声调度调节,这些语义先验在不断演变。关键的是,我们引入了一种随机混合模态运输策略,无需使用重的对比双编码器即可对齐不同的模态。我们的方法在标准掩码建模上表现出更稳定的性能,建立了一种新的可扩展、统一的文本-图像生成范式。

英文摘要:

The bifurcation of generative modeling into autoregressive approaches for discrete data (text) and diffusion approaches for continuous data (images) hinders the development of truly unified multimodal systems. While Masked Language Models (MLMs) offer efficient bidirectional context, they traditionally lack the generative fidelity of autoregressive models and the semantic continuity of diffusion models. Furthermore, extending masked generation to multimodal settings introduces severe alignment challenges and training instability. In this work, we propose \textbf{CoM-DAD} (\textbf{Co}upled \textbf{M}anifold \textbf{D}iscrete \textbf{A}bsorbing \textbf{D}iffusion), a novel probabilistic framework that reformulates multimodal generation as a hierarchical dual-process. CoM-DAD decouples high-level semantic planning from low-level token synthesis. First, we model the semantic manifold via a continuous latent diffusion process; second, we treat token generation as a discrete absorbing diffusion process, regulated by a \textbf{Variable-Rate Noise Schedule}, conditioned on these evolving semantic priors. Crucially, we introduce a \textbf{Stochastic Mixed-Modal Transport} strategy that aligns disparate modalities without requiring heavy contrastive dual-encoders. Our method demonstrates superior stability over standard masked modeling, establishing a new paradigm for scalable, unified text-image generation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑