arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.04339cs.CVcs.AI
收藏

通过弱项针对性后训练实现统一的文本图像生成

Unified Text-Image Generation with Weakness-Targeted Post-Training

FAIR at Meta(Meta旗下的FAIR);University of Texas, Austin(德克萨斯大学奥斯汀分校);University of Washington(华盛顿大学);University of California, Los Angeles(加州大学洛杉矶分校)

Jiahui Chen, Philippe Hansen-Estruch, Xiaochuang Han, Yushi Hu, Emily Dinan, Amita Kamath, Michal Drozdzal, Reyhane Askari-Hemmat, Luke Zettlemoyer, Marjan Ghazvininejad

AI总结:

本文提出通过针对性后训练实现统一文本图像生成,提升多模态生成性能。

摘要:

近年来,能够同时生成文本和图像的多模态生成架构逐渐成为文本到图像(T2I)合成的有前景方向。然而,许多现有系统依赖显式的模态切换,先生成推理文本,再手动切换到图像生成。这种分离的顺序推理过程限制了跨模态耦合,并禁止了自动多模态生成。本文探讨后训练以实现完全统一的文本图像生成,其中模型在单一推理过程中自主从文本推理过渡到视觉合成。我们研究了联合文本图像生成对T2I性能的影响以及后训练过程中每种模态的相对重要性。我们还探讨了不同的后训练数据策略,表明针对特定限制的定制数据集在性能上优于广泛图像-描述语料库或基准对齐数据。使用离线、基于奖励的后训练与完全自动生成的合成数据,我们的方法在四个多样的T2I基准上实现了多模态图像生成的改进,展示了对双模态进行奖励加权以及精心设计的后训练数据的战略价值。

英文摘要:

Unified multimodal generation architectures that jointly produce text and images have recently emerged as a promising direction for text-to-image (T2I) synthesis. However, many existing systems rely on explicit modality switching, generating reasoning text before switching manually to image generation. This separate, sequential inference process limits cross-modal coupling and prohibits automatic multimodal generation. This work explores post-training to achieve fully unified text-image generation, where a model autonomously transitions from textual reasoning to visual synthesis within a single inference process. We study this on BAGEL, a 14B mixture-of-transformers model that pairs autoregressive text generation with flow-matching image synthesis. We examine the impact of joint text-image generation on T2I performance and the relative importance of each modality during post-training. We additionally explore different post-training data strategies, showing that a targeted dataset addressing specific limitations achieves superior results compared to broad image-caption corpora or benchmark-aligned data. Using offline, reward-weighted post-training with fully self-generated synthetic data, our approach enables improvements in multimodal image generation across four diverse, independent T2I benchmarks, demonstrating the effectiveness of reward-weighting both modalities and strategically designed post-training data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑