Lumina-mGPT：通过多模态生成预训练实现灵活逼真的文本到图像生成
Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining
AI总结：
Lumina-mGPT通过多模态生成预训练和灵活渐进式监督微调，实现了高效灵活的逼真文本到图像生成，并探索了统一多模态通才的潜力。
摘要：
我们提出了Lumina-mGPT，一个多模态自回归模型家族，能够执行各种视觉和语言任务，尤其在从文本描述生成灵活逼真的图像方面表现出色。通过从多模态生成预训练（mGPT）初始化，我们证明了仅解码器的自回归（AR）模型可以通过灵活渐进式监督微调（FP-SFT）实现与现代扩散模型相当的高效图像生成性能。配备我们提出的无歧义图像表示（UniRep），Lumina-mGPT可以灵活生成不同宽高比的高质量图像。基于强大的图像生成能力，我们进一步探索了全能监督微调（Omni-SFT），这是将Lumina-mGPT提升为统一多模态通才的初步尝试。由此产生的模型展示了多样的多模态能力，包括文本到图像/多视图生成和可控生成等视觉生成任务，分割和深度估计等视觉识别任务，以及多轮视觉问答等视觉语言任务，显示了该技术方向的巨大潜力。代码和检查点可在https://github.com/Alpha-VLLM/Lumina-mGPT获取。
英文摘要：
We present Lumina-mGPT, a family of multimodal autoregressive models capable of various vision and language tasks, particularly excelling in generating flexible photorealistic images from text descriptions. By initializing from multimodal Generative PreTraining (mGPT), we demonstrate that decoder-only Autoregressive (AR) model can achieve image generation performance comparable to modern diffusion models with high efficiency through Flexible Progressive Supervised Fine-tuning (FP-SFT). Equipped with our proposed Unambiguous image Representation (UniRep), Lumina-mGPT can flexibly generate high-quality images of varying aspect ratios. Building on the strong image generation capabilities, we further explore Ominiponent Supervised Fine-tuning (Omni-SFT), an initial attempt to elevate Lumina-mGPT into a unified multi-modal generalist. The resulting model demonstrates versatile multimodal capabilities, including visual generation tasks like text-to-image/multiview generation and controllable generation, visual recognition tasks like segmentation and depth estimation, and vision-language tasks like multi-turn visual question answering, showing the rosy potential of the technical direction. Codes and checkpoints are available at https://github.com/Alpha-VLLM/Lumina-mGPT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
