arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.14289cs.LGcs.CL
收藏

TinyLLaVA:一个小规模大型多模态模型框架

TinyLLaVA: A Framework of Small-scale Large Multimodal Models

Beihang University(北京航空航天大学);Tsinghua University(清华大学)

Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, Lei Huang

AI总结:

TinyLLaVA框架通过系统研究视觉编码器、连接模块、语言模型、训练数据与策略,证明高质量数据和训练方法可使小规模多模态模型达到甚至超越更大模型性能,并开源模型与代码。

摘要:

我们提出了TinyLLaVA框架,为设计和分析小规模大型多模态模型(LMMs)提供了统一视角。我们实证研究了不同视觉编码器、连接模块、语言模型、训练数据和训练策略的影响。大量实验表明,更高质量的数据结合更好的训练策略,可以使较小的LMMs持续取得与较大LMMs相当的性能。在该框架下,我们训练了一系列小规模LMMs。我们的最佳模型TinyLLaVA-3.1B在整体性能上优于现有的7B模型,如LLaVA-1.5和Qwen-VL。我们希望这些发现能够在数据扩展、训练设置和模型选择方面为未来研究提供基线。我们的模型权重和代码将公开。

英文摘要:

We present the TinyLLaVA framework that provides a unified perspective in designing and analyzing the small-scale Large Multimodal Models (LMMs). We empirically study the effects of different vision encoders, connection modules, language models, training data and training recipes. Our extensive experiments showed that better quality of data combined with better training recipes, smaller LMMs can consistently achieve on-par performances compared to bigger LMMs. Under our framework, we train a family of small-scale LMMs. Our best model, TinyLLaVA-3.1B, achieves better overall performance against existing 7B models such as LLaVA-1.5 and Qwen-VL. We hope our findings can serve as baselines for future research in terms of data scaling, training setups and model selections. Our model weights and codes will be made public.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑