arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.18262cs.CV
收藏

MammothModa2: 一种用于多模态理解和生成的统一AR-扩散框架

MammothModa2: A Unified AR-Diffusion Framework for Multimodal Understanding and Generation

ByteDance(字节跳动)

Tao Shen, Xin Wan, Taicai Chen, Rui Zhang, Junwen Pan, Dawei Lu, Fanding Lei, Zhilin Lu, Yunfei Yang, Chen Cheng, Qi She, Chang Liu, Zhenbang Sun

AI总结:

MammothModa2通过统一的AR-扩散框架实现多模态理解和生成,结合自回归语义规划与扩散生成,取得文本到图像和指令编辑的优异表现。

摘要:

统一的多模态模型旨在在一个框架内整合理解和生成,但弥合离散语义推理与高保真视觉合成之间的差距仍然具有挑战性。我们提出了MammothModa2(Mammoth2),一种统一的自回归-扩散(AR-Diffusion)框架,旨在有效结合自回归语义规划与基于扩散的生成。Mammoth2采用串行设计:一个配备生成专家的AR路径对离散标记进行全局语义建模,而一个单流扩散变换器(DiT)解码器处理高保真的图像合成。精心设计的AR-扩散特征对齐模块结合了多层特征聚合、统一条件编码和上下文条件化,以稳定地将AR的表示与扩散解码器的连续潜在空间对齐。Mammoth2通过联合下一个标记预测和流匹配目标进行端到端训练,随后进行监督微调和强化学习,涵盖生成和编辑。在大约6000万监督生成样本和不依赖预训练生成器的情况下,Mammoth2在公共基准上实现了强大的文本到图像和基于指令的编辑性能,分别达到GenEval 0.87、DPGBench 87.2和ImgEdit 4.06,同时在多模态理解任务上与仅具备理解能力的backbone(如Qwen3-VL-8B)具有竞争力。这些结果表明,精心耦合的AR-扩散架构可以在单一参数和数据高效的模型中提供高保真的生成和编辑,同时保持强大的多模态理解能力。

英文摘要:

Unified multimodal models aim to integrate understanding and generation within a single framework, yet bridging the gap between discrete semantic reasoning and high-fidelity visual synthesis remains challenging. We present MammothModa2 (Mammoth2), a unified autoregressive-diffusion (AR-Diffusion) framework designed to effectively couple autoregressive semantic planning with diffusion-based generation. Mammoth2 adopts a serial design: an AR path equipped with generation experts performs global semantic modeling over discrete tokens, while a single-stream Diffusion Transformer (DiT) decoder handles high-fidelity image synthesis. A carefully designed AR-Diffusion feature alignment module combines multi-layer feature aggregation, unified condition encoding, and in-context conditioning to stably align AR's representations with the diffusion decoder's continuous latents. Mammoth2 is trained end-to-end with joint Next-Token Prediction and Flow Matching objectives, followed by supervised fine-tuning and reinforcement learning over both generation and editing. With roughly 60M supervised generation samples and no reliance on pre-trained generators, Mammoth2 delivers strong text-to-image and instruction-based editing performance on public benchmarks, achieving 0.87 on GenEval, 87.2 on DPGBench, and 4.06 on ImgEdit, while remaining competitive with understanding-only backbones (e.g., Qwen3-VL-8B) on multimodal understanding tasks. These results suggest that a carefully coupled AR-Diffusion architecture can provide high-fidelity generation and editing while maintaining strong multimodal comprehension within a single, parameter- and data-efficient model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑