使用MetaQueries进行模态间迁移
Transfer between Modalities with MetaQueries
AI总结：
本文提出MetaQueries可学习查询，作为多模态大语言模型与扩散模型之间的高效接口，在冻结MLLM主干的情况下实现知识增强图像生成，并支持图像编辑等高级应用。
摘要：
统一多模态模型旨在整合理解（文本输出）与生成（像素输出），但在单一架构中对齐这些不同模态通常需要复杂的训练方案和精细的数据平衡。我们提出了MetaQueries，这是一组可学习的查询，充当自回归多模态大语言模型（MLLMs）与扩散模型之间的高效接口。MetaQueries将MLLM的潜在表示连接到扩散解码器，通过利用MLLM的深度理解和推理能力实现知识增强的图像生成。我们的方法简化了训练，仅需成对的图像-标题数据和标准的扩散目标。值得注意的是，即使MLLM主干保持冻结，这种迁移仍然有效，从而在保持其最先进的多模态理解能力的同时实现强大的生成性能。此外，我们的方法灵活，可以轻松进行指令微调，以用于图像编辑和主体驱动生成等高级应用。
英文摘要：
Unified multimodal models aim to integrate understanding (text output) and generation (pixel output), but aligning these different modalities within a single architecture often demands complex training recipes and careful data balancing. We introduce MetaQueries, a set of learnable queries that act as an efficient interface between autoregressive multimodal LLMs (MLLMs) and diffusion models. MetaQueries connects the MLLM's latents to the diffusion decoder, enabling knowledge-augmented image generation by leveraging the MLLM's deep understanding and reasoning capabilities. Our method simplifies training, requiring only paired image-caption data and standard diffusion objectives. Notably, this transfer is effective even when the MLLM backbone remains frozen, thereby preserving its state-of-the-art multimodal understanding capabilities while achieving strong generative performance. Additionally, our method is flexible and can be easily instruction-tuned for advanced applications such as image editing and subject-driven generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
