TBAC-UniImage:通过阶梯侧扩散调优实现统一理解与生成
TBAC-UniImage: Unified Understanding and Generation by Ladder-Side Diffusion Tuning
AI总结:
针对现有扩散基统一模型的浅连接或高成本问题,TBAC-UniImage通过将预训练扩散模型(生成阶梯)与MLLM深度集成,利用MLLM多层表示作为生成条件,实现更深度细粒度的多模态理解与生成统一。
摘要:
本文介绍了TBAC-UniImage,一种用于多模态理解与生成的新型统一模型。我们通过将预训练扩散模型(作为生成阶梯)与多模态大语言模型(MLLM)深度集成来实现这一点。此前基于扩散的统一模型面临两个主要限制:一种方法仅将MLLM的最终隐藏状态作为生成条件,这种浅连接使生成器与MLLM中间层丰富的分层表示隔离;另一种方法从头预训练统一生成架构,计算成本高昂且对多数研究者不现实。为克服这些问题,我们探索新范式:不再依赖单一输出,而是使用MLLM多个不同层的表示作为扩散模型的生成条件,将预训练生成器视为接收MLLM理解过程各深度指导的阶梯,从而实现理解与生成更深度、更细粒度的统一。
英文摘要:
This paper introduces TBAC-UniImage, a novel unified model for multimodal understanding and generation. We achieve this by deeply integrating a pre-trained Diffusion Model, acting as a generative ladder, with a Multimodal Large Language Model (MLLM). Previous diffusion-based unified models face two primary limitations. One approach uses only the MLLM's final hidden state as the generative condition. This creates a shallow connection, as the generator is isolated from the rich, hierarchical representations within the MLLM's intermediate layers. The other approach, pretraining a unified generative architecture from scratch, is computationally expensive and prohibitive for many researchers. To overcome these issues, our work explores a new paradigm. Instead of relying on a single output, we use representations from multiple, diverse layers of the MLLM as generative conditions for the diffusion model. This method treats the pre-trained generator as a ladder, receiving guidance from various depths of the MLLM's understanding process. Consequently, TBAC-UniImage achieves a much deeper and more fine-grained unification of understanding and generation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。