CoDi-2：上下文内、交错式与交互式的任意到任意生成
CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation
AI总结：
CoDi-2 是一个多模态大语言模型，通过模态与语言对齐实现任意到任意生成，并具备上下文内学习、推理和交互式多轮对话能力，在图像生成、视觉变换和音频编辑等任务上超越领域专用模型。
摘要：
我们提出 CoDi-2，这是一种通用且可交互的多模态大语言模型（MLLM），能够在任意到任意的输入输出模态范式中遵循复杂的多模态交错指令、进行上下文内学习（ICL）、推理、对话、编辑等。通过在编码和生成两方面将模态与语言对齐，CoDi-2 使大语言模型（LLM）不仅能够理解复杂的模态交错指令和上下文内示例，还能够在连续特征空间中自回归地生成有依据且连贯的多模态输出。为了训练 CoDi-2，我们构建了一个大规模生成数据集，涵盖文本、视觉和音频中的上下文内多模态指令。CoDi-2 在多模态生成方面展示了广泛的零样本能力，例如通过多轮交互式对话实现上下文内学习、推理以及任意到任意模态生成的组合性。CoDi-2 在主体驱动的图像生成、视觉变换和音频编辑等任务上超越了以往的领域专用模型。CoDi-2 标志着在开发一个全面的多模态基础模型方面取得了重大突破，该模型擅长解释上下文内的语言-视觉-音频交错指令并产生多模态输出。
英文摘要：
We present CoDi-2, a versatile and interactive Multimodal Large Language Model (MLLM) that can follow complex multimodal interleaved instructions, conduct in-context learning (ICL), reason, chat, edit, etc., in an any-to-any input-output modality paradigm. By aligning modalities with language for both encoding and generation, CoDi-2 empowers Large Language Models (LLMs) to not only understand complex modality-interleaved instructions and in-context examples, but also autoregressively generate grounded and coherent multimodal outputs in the continuous feature space. To train CoDi-2, we build a large-scale generation dataset encompassing in-context multimodal instructions across text, vision, and audio. CoDi-2 demonstrates a wide range of zero-shot capabilities for multimodal generation, such as in-context learning, reasoning, and compositionality of any-to-any modality generation through multi-round interactive conversation. CoDi-2 surpasses previous domain-specific models on tasks such as subject-driven image generation, vision transformation, and audio editing. CoDi-2 signifies a substantial breakthrough in developing a comprehensive multimodal foundation model adept at interpreting in-context language-vision-audio interleaved instructions and producing multimodal outputs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
