联合训练大型自回归多模态模型
Jointly Training Large Autoregressive Multimodal Models
AI总结：
针对文本与图像模态难以统一生成的问题，本文提出JAM模块化框架融合现有生成模型，并采用数据高效的混合模态指令微调策略，实现首个专门面向高质量无缝多模态输出的模型。
摘要：
近年来，语言模型与文本到图像模型的大规模预训练进展彻底改变了机器学习领域。然而，将这两种模态整合到一个能够生成无缝多模态输出的稳健单一模型中，仍是一项重大挑战。为弥补这一差距，我们提出 Joint Autoregressive Mixture（JAM）框架，这是一种模块化方法，可系统性地融合现有文本生成模型与图像生成模型。我们还介绍了一种专门为混合模态生成任务定制、数据高效的指令微调策略。最终的指令微调模型在生成高质量多模态输出方面展现出无与伦比的性能，并代表了首个明确为此目的设计的模型。
英文摘要：
In recent years, advances in the large-scale pretraining of language and text-to-image models have revolutionized the field of machine learning. Yet, integrating these two modalities into a single, robust model capable of generating seamless multimodal outputs remains a significant challenge. To address this gap, we present the Joint Autoregressive Mixture (JAM) framework, a modular approach that systematically fuses existing text and image generation models. We also introduce a specialized, data-efficient instruction-tuning strategy, tailored for mixed-modal generation tasks. Our final instruct-tuned model demonstrates unparalleled performance in generating high-quality multimodal outputs and represents the first model explicitly designed for this purpose.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
