NaViL：数据约束下原生多模态大语言模型缩放性质的重思考
NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints
AI总结：
本文在数据约束下系统研究原生多模态大语言模型的端到端训练设计空间与缩放性质，提出最优元架构及NaViL模型，在14个基准上验证了其竞争性能，并揭示视觉编码器与LLM的正相关缩放关系。
摘要：
组合式训练一直是现有多模态大语言模型（MLLMs）中的事实标准范式，即通过连续的多模态预训练将预训练的视觉编码器与预训练的LLMs连接起来。然而，由于训练分离，该范式的多模态缩放性质难以探索。在本文中，我们专注于以端到端方式对MLLMs进行原生训练，并在实际设置（即数据约束）下系统地研究其设计空间和缩放性质。通过对MLLM中各种选择的仔细研究，我们获得了在性能和训练成本之间最佳平衡的最优元架构。之后，我们进一步探索原生MLLM的缩放性质，并指出视觉编码器与LLMs之间存在正相关的缩放关系。基于这些发现，我们提出了一种名为NaViL的原生MLLM，并结合了一种简单且成本效益高的方案。在14个多模态基准上的实验结果证实了NaViL相对于现有MLLMs的竞争性能。此外，我们的发现和结果为未来原生MLLM的研究提供了深入的见解。
英文摘要：
Compositional training has been the de-facto paradigm in existing Multimodal Large Language Models (MLLMs), where pre-trained vision encoders are connected with pre-trained LLMs through continuous multimodal pre-training. However, the multimodal scaling property of this paradigm remains difficult to explore due to the separated training. In this paper, we focus on the native training of MLLMs in an end-to-end manner and systematically study its design space and scaling property under a practical setting, i.e., data constraint. Through careful study of various choices in MLLM, we obtain the optimal meta-architecture that best balances performance and training cost. After that, we further explore the scaling properties of the native MLLM and indicate the positively correlated scaling relationship between visual encoders and LLMs. Based on these findings, we propose a native MLLM called NaViL, combined with a simple and cost-effective recipe. Experimental results on 14 multimodal benchmarks confirm the competitive performance of NaViL against existing MLLMs. Besides that, our findings and results provide in-depth insights for the future study of native MLLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
