多模态表示对齐用于图像生成：文本-图像交错控制比想象中更容易
Multimodal Representation Alignment for Image Generation: Text-Image Interleaved Control Is Easier Than You Think
AI总结：
针对任意文本-图像交错控制缺失问题，提出Dream Engine框架，利用LMM共享表示空间，通过两阶段训练在GenEval上达到0.69分，媲美SD3.5和FLUX。
摘要：
先进文本到图像生成领域正涌现出统一框架，这些框架将强大的文本编码器（如CLIP和T5）与扩散Transformer骨干网络相结合。尽管已有尝试通过额外条件（如Canny边缘和深度图）控制输出图像，但一个支持任意文本-图像交错控制的全面框架仍然缺失。这一差距在生成过程中尝试合并多张图像的概念或视觉元素时尤为明显。为弥补这一差距，我们进行了初步实验，表明大型多模态模型（LMMs）提供了有效的共享表示空间，其中图像和文本能够良好对齐，可作为外部扩散模型的条件。基于这一发现，我们提出了Dream Engine，一个高效且统一的框架，专为图像生成模型中的任意文本-图像交错控制而设计。在SD3.5等强大文本到图像模型的基础上，我们通过引入如QwenVL等通用多模态信息编码器，替换了原有的纯文本编码器。我们的方法采用两阶段训练范式，包括联合文本-图像对齐和多模态交错指令微调。实验表明，该训练方法有效，在GenEval基准上取得了0.69的总体得分，与SD3.5和FLUX等最先进的文本到图像模型性能相当。
英文摘要：
The field of advanced text-to-image generation is witnessing the emergence of unified frameworks that integrate powerful text encoders, such as CLIP and T5, with Diffusion Transformer backbones. Although there have been efforts to control output images with additional conditions, like canny and depth map, a comprehensive framework for arbitrary text-image interleaved control is still lacking. This gap is especially evident when attempting to merge concepts or visual elements from multiple images in the generation process. To mitigate the gap, we conducted preliminary experiments showing that large multimodal models (LMMs) offer an effective shared representation space, where image and text can be well-aligned to serve as a condition for external diffusion models. Based on this discovery, we propose Dream Engine, an efficient and unified framework designed for arbitrary text-image interleaved control in image generation models. Building on powerful text-to-image models like SD3.5, we replace the original text-only encoders by incorporating versatile multimodal information encoders such as QwenVL. Our approach utilizes a two-stage training paradigm, consisting of joint text-image alignment and multimodal interleaved instruction tuning. Our experiments demonstrate that this training method is effective, achieving a 0.69 overall score on the GenEval benchmark, and matching the performance of state-of-the-art text-to-image models like SD3.5 and FLUX.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
