arXiv 2507.095742026-05-28cs.CVcs.AIcs.CL

MENTOR: 面向自回归视觉生成模型的高效多模态条件微调

MENTOR: Efficient Multimodal-Conditioned Tuning for Autoregressive Vision Generation Models

University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Tsinghua University（清华大学）；Peking University（北京大学）；Microsoft（微软公司）

Haozhe Zhao, Zefan Cai, Shuzheng Si, Liang Chen, Jiuxiang Gu, Wen Xiao, Minjia Zhang, Junjie Hu

备注：: Findings of ACL 2026

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出MENTOR框架，通过两阶段训练范式实现自回归图像生成器与多模态输入的细粒度token级对齐，无需辅助适配器或交叉注意力模块，在DreamBench++上取得优异性能。

摘要：

最近的文本到图像模型能够生成高质量结果，但在精确视觉控制、平衡多模态输入以及需要大量训练以实现复杂多模态图像生成方面仍存在困难。为解决这些局限，我们提出MENTOR，一种新颖的自回归（AR）框架，用于高效的多模态条件微调以实现自回归多模态图像生成。MENTOR将AR图像生成器与两阶段训练范式相结合，无需依赖辅助适配器或交叉注意力模块，即可实现多模态输入与图像输出之间的细粒度、token级对齐。两阶段训练包括：（1）多模态对齐阶段，建立稳健的像素级和语义级对齐；随后是（2）多模态指令微调阶段，平衡多模态输入的整合并增强生成可控性。尽管模型规模适中、基础组件非最优且训练资源有限，MENTOR在DreamBench++基准测试上仍取得了强劲性能，在概念保持和提示遵循方面优于竞争基线。此外，与基于扩散的方法相比，我们的方法具有更优的图像重建保真度、广泛的任务适应性以及更高的训练效率。数据集、代码和模型可在 https://github.com/HaozheZhao/MENTOR 获取。

英文摘要：

Recent text-to-image models produce high-quality results but still struggle with precise visual control, balancing multimodal inputs, and requiring extensive training for complex multimodal image generation. To address these limitations, we propose MENTOR, a novel autoregressive (AR) framework for efficient Multimodal-conditioned Tuning for Autoregressive multimodal image generation. MENTOR combines an AR image generator with a two-stage training paradigm, enabling fine-grained, token-level alignment between multimodal inputs and image outputs without relying on auxiliary adapters or cross-attention modules. The two-stage training consists of: (1) a multimodal alignment stage that establishes robust pixel- and semantic-level alignment, followed by (2) a multimodal instruction tuning stage that balances the integration of multimodal inputs and enhances generation controllability. Despite modest model size, suboptimal base components, and limited training resources, MENTOR achieves strong performance on the DreamBench++ benchmark, outperforming competitive baselines in concept preservation and prompt following. Additionally, our method delivers superior image reconstruction fidelity, broad task adaptability, and improved training efficiency compared to diffusion-based methods. Dataset, code, and models are available at: https://github.com/HaozheZhao/MENTOR

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。