arXiv 2311.075752023-11-13cs.CVcs.AIcs.CLcs.LG

SPHINX：多模态大语言模型中权重、任务与视觉嵌入的联合混合

SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models

Shanghai AI Laboratory（上海人工智能实验室）；MMLab, CUHK（香港中文大学多媒体实验室）；ShanghaiTech University（上海科技大学）

Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, Jiaming Han, Siyuan Huang, Yichi Zhang, Xuming H… 展开作者

Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, Jiaming Han, Siyuan Huang, Yichi Zhang, Xuming He, Hongsheng Li, Yu Qiao

备注：: Work in progress. Code and demos are released at https://github.com/Alpha-VLLM/LLaMA2-Accessory

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出SPHINX多模态大语言模型，通过联合混合模型权重、任务和视觉嵌入，实现更强的视觉-语言对齐与多功能理解，并在高分辨率图像解析上取得卓越性能。

摘要：

我们提出了SPHINX，一个具有模型权重、调优任务和视觉嵌入联合混合的多功能多模态大语言模型（MLLM）。首先，为了更强的视觉-语言对齐，我们在预训练期间解冻大语言模型（LLM），并引入一种在真实世界和合成数据训练的LLM之间的权重混合策略。通过直接整合来自两个领域的权重，混合后的LLM能够高效地融合多样化的语义，并具有良好的鲁棒性。然后，为了实现多用途能力，我们混合了多种任务进行联合视觉指令调优，并设计了任务特定的指令以避免任务间冲突。除了基本的视觉问答，我们还包含了更具挑战性的任务，如区域级理解、标题定位、文档布局检测和人体姿态估计，这些任务在不同场景下相互促进。此外，我们提出从不同的网络架构、预训练范式和信息粒度中提取全面的视觉嵌入，为语言模型提供更鲁棒的图像表示。基于我们提出的联合混合，SPHINX在广泛的应用中展现出优越的多模态理解能力。在此基础上，我们进一步提出了一种高效策略，旨在更好地捕捉高分辨率图像的细粒度外观。通过不同尺度和高分辨率子图像的混合，SPHINX在现有评估基准上取得了卓越的视觉解析和推理性能。我们希望我们的工作能为未来MLLM研究中的联合混合探索提供启示。代码已在https://github.com/Alpha-VLLM/LLaMA2-Accessory发布。

英文摘要：

We present SPHINX, a versatile multi-modal large language model (MLLM) with a joint mixing of model weights, tuning tasks, and visual embeddings. First, for stronger vision-language alignment, we unfreeze the large language model (LLM) during pre-training, and introduce a weight mix strategy between LLMs trained by real-world and synthetic data. By directly integrating the weights from two domains, the mixed LLM can efficiently incorporate diverse semantics with favorable robustness. Then, to enable multi-purpose capabilities, we mix a variety of tasks for joint visual instruction tuning, and design task-specific instructions to avoid inter-task conflict. In addition to the basic visual question answering, we include more challenging tasks such as region-level understanding, caption grounding, document layout detection, and human pose estimation, contributing to mutual enhancement over different scenarios. Additionally, we propose to extract comprehensive visual embeddings from various network architectures, pre-training paradigms, and information granularity, providing language models with more robust image representations. Based on our proposed joint mixing, SPHINX exhibits superior multi-modal understanding capabilities on a wide range of applications. On top of this, we further propose an efficient strategy aiming to better capture fine-grained appearances of high-resolution images. With a mixing of different scales and high-resolution sub-images, SPHINX attains exceptional visual parsing and reasoning performance on existing evaluation benchmarks. We hope our work may cast a light on the exploration of joint mixing in future MLLM research. Code is released at https://github.com/Alpha-VLLM/LLaMA2-Accessory.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。