指令微调大型多模态模型扩展性的实证研究
An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
AI总结：
本文实证研究将 LLaVA 扩展至 33B 和 65B/70B，发现扩展规模、高分辨率图像及多模态-语言数据混合能持续提升模型性能，且 LoRA/QLoRA 参数高效微调可与全模型微调相媲美。
摘要：
视觉指令微调近期在开源大型多模态模型（LMM）如 LLaVA 和 MiniGPT-4 上展现出令人鼓舞的进展。然而，现有大多数开源 LMM 研究使用的是参数量不超过 13B 的模型。本文对将 LLaVA 扩展至 33B 和 65B/70B 进行了实证研究，并分享了我们在图像分辨率、数据混合以及 LoRA/QLoRA 等参数高效训练方法方面的探索发现。这些因素通过其在真实世界任务中完成多模态和语言能力的影响进行评估。我们发现，扩展 LMM 能够持续提升模型性能并改善语言能力，且对 LMM 进行 LoRA/QLoRA 微调的性能与全模型微调的性能相当。此外，研究强调了更高图像分辨率以及混合多模态-语言数据对提升 LMM 性能的重要性，并且视觉指令微调有时也能改善 LMM 的纯语言能力。我们希望本研究能让更大规模的先进 LMM 研究更易于开展，从而为未来研究建立更强的基线。代码和检查点将公开发布。
英文摘要：
Visual instruction tuning has recently shown encouraging progress with open-source large multimodal models (LMM) such as LLaVA and MiniGPT-4. However, most existing studies of open-source LMM are performed using models with 13B parameters or smaller. In this paper we present an empirical study of scaling LLaVA up to 33B and 65B/70B, and share our findings from our explorations in image resolution, data mixing and parameter-efficient training methods such as LoRA/QLoRA. These are evaluated by their impact on the multi-modal and language capabilities when completing real-world tasks in the wild. We find that scaling LMM consistently enhances model performance and improves language capabilities, and performance of LoRA/QLoRA tuning of LMM are comparable to the performance of full-model fine-tuning. Additionally, the study highlights the importance of higher image resolutions and mixing multimodal-language data to improve LMM performance, and visual instruction tuning can sometimes improve LMM's pure language capability. We hope that this study makes state-of-the-art LMM research at a larger scale more accessible, thus helping establish stronger baselines for future research. Code and checkpoints will be made public.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
