arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.17080cs.CV
收藏

将统一多模态模型扩展至长时序交错生成

Scaling Unified Multimodal Models to Long-Horizon Interleaved Generation

Renmin University of China(中国人民大学);Tencent(腾讯);Tsinghua University(清华大学)
Ziqiao Peng, Yutao Cui, Wang Lin, Zhong-Yu Li, Qingwen Gu, Tianhui Song, Chengyou Jia, Guozhen Zhang, Xuerui Qiu, Qixun Wang, Miles Yang, Zhao Zhong, Yonglong T… 展开作者

Ziqiao Peng, Yutao Cui, Wang Lin, Zhong-Yu Li, Qingwen Gu, Tianhui Song, Chengyou Jia, Guozhen Zhang, Xuerui Qiu, Qixun Wang, Miles Yang, Zhao Zhong, Yonglong Tian, Hongyan Liu, Jun He

AI总结:

本研究针对统一多模态模型长时序交错生成的挑战,提出视觉记忆框架并构建LongMIG-Bench基准,实现了高保真、角色与场景一致性强的长时序交错生成。

摘要:

统一多模态模型(UMMs)为交错文本-图像生成提供了接口,生成的视觉状态可直接为后续推理与合成提供信息。然而,现有系统主要在短轨迹或简化视觉内容上进行演示。我们开发了一种原生在交错多模态数据上训练的大型UMM,具备强大的图像合成、文本渲染、复杂构图及多模态规划能力,并研究如何将这些原生能力扩展至更长的自回归轨迹。主要挑战在于生成的图像会在上下文持续累积:无关或不完美的视觉状态会干扰后续合成,而长时间间隔后仍需恢复已建立的角色与场景。我们引入视觉记忆框架,该框架可维持持久的视觉参考,选择性检索与场景相关的历史状态,同时保留完整的语义历史,并通过少步自回滚使模型适应不完美的生成历史。我们进一步推出LongMIG-Bench,用于评估长时序文本到多图像及图像到多图像的生成。我们的模型实现了高保真的长时序交错生成,具备强大的角色与场景一致性,证明了视觉记忆可将原生交错生成长时序扩展至更长轨迹。

英文摘要:

Unified multimodal models (UMMs) provide an interface for interleaved text-image generation, where generated visual states can directly inform subsequent reasoning and synthesis. However, existing systems are still primarily demonstrated on short trajectories or simplified visual content. We develop a large UMM natively trained on interleaved multimodal data, with strong capabilities in image synthesis, text rendering, complex composition, and multimodal planning, and study how to extend these native capabilities to substantially longer autoregressive trajectories. The main challenge is that generated images continuously accumulate in context: irrelevant or imperfect visual states interfere with future synthesis, while previously established characters and scenes must still be recovered after long intervals. We introduce a visual-memory framework that maintains persistent visual references, selectively retrieves scene-relevant historical states while preserving the complete semantic history, and uses few-step self-rollout to adapt the model to imperfect generated histories. We further introduce LongMIG-Bench for evaluating long-horizon text-to-multi-image and image-to-multi-image generation. Our model achieves high-fidelity long-horizon interleaved generation with strong character and scene consistency, demonstrating how visual memory can scale native interleaved generation to substantially longer trajectories.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑