ILLUME：照亮你的大语言模型，使其具备视觉、绘图与自增强能力
ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance
AI总结：
本文提出统一多模态大语言模型ILLUME，通过统一下一词元预测整合理解与生成能力，借助语义视觉分词器和多阶段训练提升数据效率，还提出自增强对齐方案促进能力协同，在多项基准上性能达顶尖水平。
摘要：
本文提出了ILLUME，这是一种统一的多模态大语言模型（MLLM），通过统一的下一词元预测范式，在单个大语言模型中无缝整合多模态理解与生成能力。为解决图文对齐通常所需的数据集规模过大的问题，我们提出通过设计融入语义信息的视觉分词器与渐进式多阶段训练流程来提升数据效率。该方法将预训练所需的数据集规模降至仅1500万，比常规需求少四倍以上，同时性能与Janus等现有统一多模态大语言模型相当甚至更优。此外，为推动理解与生成能力间的协同增强（这一点在以往研究中探索不足），我们提出了一种新颖的自增强多模态对齐方案。该方案监督多模态大语言模型自我评估文本描述与自生成图像之间的一致性，帮助模型更准确地解读图像，避免图像生成中因对齐偏差导致的不真实、错误预测。大量实验表明，我们提出的ILLUME在多模态理解、生成与编辑的各类基准测试中表现突出，可与顶尖的统一多模态大语言模型及专用模型相媲美。
英文摘要：
In this paper, we introduce ILLUME, a unified multimodal large language model (MLLM) that seamlessly integrates multimodal understanding and generation capabilities within a single large language model through a unified next-token prediction formulation. To address the large dataset size typically required for image-text alignment, we propose to enhance data efficiency through the design of a vision tokenizer that incorporates semantic information and a progressive multi-stage training procedure. This approach reduces the dataset size to just 15M for pretraining -- over four times fewer than what is typically needed -- while achieving competitive or even superior performance with existing unified MLLMs, such as Janus. Additionally, to promote synergistic enhancement between understanding and generation capabilities, which is under-explored in previous works, we introduce a novel self-enhancing multimodal alignment scheme. This scheme supervises the MLLM to self-assess the consistency between text descriptions and self-generated images, facilitating the model to interpret images more accurately and avoid unrealistic and incorrect predictions caused by misalignment in image generation. Based on extensive experiments, our proposed ILLUME stands out and competes with state-of-the-art unified MLLMs and specialized models across various benchmarks for multimodal understanding, generation, and editing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
