DeepSpeed-VisualChat：通过多模态因果注意力实现多轮多图交错对话
DeepSpeed-VisualChat: Multi-Round Multi-Image Interleave Chat via Multi-Modal Causal Attention
AI总结：
DeepSpeed-VisualChat框架通过多模态因果注意力和数据混合技术，提升大语言模型在多轮多图交错对话中的处理能力，并开源支持高达70B参数规模。
摘要：
现有的大多数多模态模型由于无法熟练处理多图、多轮对话中交错的图像与文本输入，在训练资源分配和数据可获取性方面面临显著限制，影响了它们在各种交互领域中的适应性和可扩展性。为解决这一问题，我们提出了DeepSpeed-VisualChat框架，旨在通过引入多模态能力来优化大语言模型（LLMs），重点提升大型视觉与语言模型处理交错输入的能力。我们的框架具有以下显著特点：（1）开源支持多轮和多图对话；（2）引入了一种创新的多模态因果注意力机制；（3）利用现有数据集上的数据混合技术，确保多轮、多图对话中的无缝交互。与现有框架相比，DeepSpeed-VisualChat展现出卓越的可扩展性，可支持高达70B参数的语言模型规模，代表了多模态语言模型的重大进步，并为未来探索奠定了坚实基础。
英文摘要：
Most of the existing multi-modal models, hindered by their incapacity to adeptly manage interleaved image-and-text inputs in multi-image, multi-round dialogues, face substantial constraints in resource allocation for training and data accessibility, impacting their adaptability and scalability across varied interaction realms. To address this, we present the DeepSpeed-VisualChat framework, designed to optimize Large Language Models (LLMs) by incorporating multi-modal capabilities, with a focus on enhancing the proficiency of Large Vision and Language Models in handling interleaved inputs. Our framework is notable for (1) its open-source support for multi-round and multi-image dialogues, (2) introducing an innovative multi-modal causal attention mechanism, and (3) utilizing data blending techniques on existing datasets to assure seamless interactions in multi-round, multi-image conversations. Compared to existing frameworks, DeepSpeed-VisualChat shows superior scalability up to 70B parameter language model size, representing a significant advancement in multi-modal language models and setting a solid foundation for future explorations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
