arXiv 2602.005082026-02-03cs.CV

DuoGen：迈向通用的交错多模态生成

DuoGen: Towards General Purpose Interleaved Multimodal Generation

Georgia Tech（佐治亚理工学院）；NVIDIA research.nvidia.com/labs/dir/duogen（NVIDIA 研究所）

Min Shi, Xiaohui Zeng, Jiannan Huang, Yin Cui, Francesco Ferroni, Jialuo Li, Shubham Pachori, Zhaoshuo Li, Yogesh Balaji, Haoxiang Wang, Tsung-Yi Lin, Xiao Fu, … 展开作者

Min Shi, Xiaohui Zeng, Jiannan Huang, Yin Cui, Francesco Ferroni, Jialuo Li, Shubham Pachori, Zhaoshuo Li, Yogesh Balaji, Haoxiang Wang, Tsung-Yi Lin, Xiao Fu, Yue Zhao, Chieh-Yun Chen, Ming-Yu Liu, Humphrey Shi

备注：: Technical Report. Project Page: https://research.nvidia.com/labs/dir/duogen/

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

DuoGen通过系统性地解决数据整理、架构设计和评估问题，实现了通用的交错多模态生成，提升了文本质量、图像保真度和图像-上下文对齐性能。

摘要：

交错多模态生成使模型具备超越单模态生成模型的能力，例如逐步指导说明、视觉规划以及为推理生成视觉草图。然而，现有交错生成模型在一般指令下的质量仍受到训练数据不足和基础模型容量的限制。我们提出了DuoGen，一种通用的交错生成框架，系统性地解决了数据整理、架构设计和评估问题。在数据方面，我们通过将经过整理的原始网站重新编写成多模态对话，并结合涵盖日常场景的多样化合成示例，构建了一个大规模、高质量的指令微调数据集。在架构上，DuoGen利用了预训练多模态大语言模型强大的视觉理解能力，以及在视频生成上预训练的扩散变换器（DiT）的视觉生成能力，避免了昂贵的单模态预训练，实现了灵活的基础模型选择。一种两阶段解耦策略首先对MLLM进行指令微调，然后利用经过整理的交错图像-文本序列对DiT进行对齐。在公共和新提出的基准测试中，DuoGen在文本质量、图像保真度和图像-上下文对齐方面均优于先前的开源模型，并在统一生成模型中实现了文本到图像和图像编辑方面的最先进性能。数据和代码将在https://research.nvidia.com/labs/dir/duogen/上发布。

英文摘要：

Interleaved multimodal generation enables capabilities beyond unimodal generation models, such as step-by-step instructional guides, visual planning, and generating visual drafts for reasoning. However, the quality of existing interleaved generation models under general instructions remains limited by insufficient training data and base model capacity. We present DuoGen, a general-purpose interleaved generation framework that systematically addresses data curation, architecture design, and evaluation. On the data side, we build a large-scale, high-quality instruction-tuning dataset by combining multimodal conversations rewritten from curated raw websites, and diverse synthetic examples covering everyday scenarios. Architecturally, DuoGen leverages the strong visual understanding of a pretrained multimodal LLM and the visual generation capabilities of a diffusion transformer (DiT) pretrained on video generation, avoiding costly unimodal pretraining and enabling flexible base model selection. A two-stage decoupled strategy first instruction-tunes the MLLM, then aligns DiT with it using curated interleaved image-text sequences. Across public and newly proposed benchmarks, DuoGen outperforms prior open-source models in text quality, image fidelity, and image-context alignment, and also achieves state-of-the-art performance on text-to-image and image editing among unified generation models. Data and code will be released at https://research.nvidia.com/labs/dir/duogen/.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。