We introduce Transfusion, a recipe for training a multi-modal model over discrete and continuous data. Transfusion combines the language modeling loss function (next token prediction) with diffusion to train a single transformer over mixed-modality sequences. We pretrain multiple Transfusion models up to 7B parameters from scratch on a mixture of text and image data, establishing scaling laws with respect to a variety of uni- and cross-modal benchmarks. Our experiments show that Transfusion scales significantly better than quantizing images and training a language model over discrete image tokens. By introducing modality-specific encoding and decoding layers, we can further improve the performance of Transfusion models, and even compress each image to just 16 patches. We further demonstrate that scaling our Transfusion recipe to 7B parameters and 2T multi-modal tokens produces a model that can generate images and text on a par with similar scale diffusion models and language models, reaping the benefits of both worlds.
Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spaces
Machine Learning Center, Georgia Institute of Technology, Atlanta, GA;School of Mathematics, Georgia Institute of Technology, Atlanta, GA;Department of Mathematics \& Statistics, SUNY Albany, NY
Imagine Before You Draw: Visual Prompt Engineering for Image Generation
Nanyang Technological University(南洋理工大学);National University of Singapore(国立新加坡大学);Zhejiang University(浙江大学);The Chinese University of Hong Kong(香港中文大学)
DiffStyler: Controllable Dual Diffusion for Text-Driven Image Stylization
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Kuaishou Technology(快手科技)
Diff4MMLiTS: Advanced Multimodal Liver Tumor Segmentation via Diffusion-Based Image Synthesis and Alignment
Department of Electronic and Electrical Engineering, Southern University of Science and Technology, Shenzhen, China(电子与电气工程系,南方科技大学,深圳,中国);Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong SAR, China(电气与电子工程系,香港大学,香港特别行政区,中国);Department of Radiology, Zhongda Hospital, Medical School, Southeast University, Nanjing, China(放射科,中大医院,医学院,东南大学,南京,中国);Jiaxing Research Institute, Southern University of Science and Technology, Jiaxing, China(嘉兴研究所,南方科技大学,嘉兴,中国)
Evaluating Single and Multi-Omics Based Explainable Artificial Intelligence (MOXAI) for Molecular Subclass Classification of Adult-Type Diffuse Gliomas