arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.11039cs.AIcs.CV
收藏

Transfusion:用一个多模态模型预测下一个词元并扩散生成图像

Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model

Meta;Waymo(威摩公司);University of Southern California(南加州大学)

Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, Omer Levy

AI总结:

Transfusion通过结合下一个词元预测与扩散,在单一Transformer上统一处理文本和图像,实现了7B参数下的高效多模态生成,性能媲美专用模型。

摘要:

我们引入了Transfusion,一种在离散和连续数据上训练多模态模型的方案。Transfusion将语言建模损失函数(下一个词元预测)与扩散相结合,以在混合模态序列上训练单个Transformer。我们从零开始在文本和图像数据的混合上预训练多个Transfusion模型,参数规模最高达7B,并建立了相对于多种单模态和跨模态基准的扩展定律。我们的实验表明,Transfusion的扩展性能显著优于将图像量化并在离散图像词元上训练语言模型的方法。通过引入模态特定的编码和解码层,我们可以进一步提升Transfusion模型的性能,甚至将每张图像压缩至仅16个图像块。我们进一步证明,将我们的Transfusion方案扩展到7B参数和2T多模态词元,可以产生一个在图像和文本生成方面与类似规模的扩散模型和语言模型相当,同时兼得两者优势的模型。

英文摘要:

We introduce Transfusion, a recipe for training a multi-modal model over discrete and continuous data. Transfusion combines the language modeling loss function (next token prediction) with diffusion to train a single transformer over mixed-modality sequences. We pretrain multiple Transfusion models up to 7B parameters from scratch on a mixture of text and image data, establishing scaling laws with respect to a variety of uni- and cross-modal benchmarks. Our experiments show that Transfusion scales significantly better than quantizing images and training a language model over discrete image tokens. By introducing modality-specific encoding and decoding layers, we can further improve the performance of Transfusion models, and even compress each image to just 16 patches. We further demonstrate that scaling our Transfusion recipe to 7B parameters and 2T multi-modal tokens produces a model that can generate images and text on a par with similar scale diffusion models and language models, reaping the benefits of both worlds.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑