arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.20566cs.CVcs.CLcs.LG
收藏

MM1.5:多模态大语言模型微调的方法、分析与见解

MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning

Apple(苹果公司)
Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang, Dhruti Shah, Xianzhi Du, Bowen Zhang, Yanghao Li, Sam Dodge, Keen You, Zhen Yan… 展开作者

Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang, Dhruti Shah, Xianzhi Du, Bowen Zhang, Yanghao Li, Sam Dodge, Keen You, Zhen Yang, Aleksei Timofeev, Mingze Xu, Hong-You Chen, Jean-Philippe Fauconnier, Zhengfeng Lai, Haoxuan You, Zirui Wang, Afshin Dehghan, Peter Grasch, Yinfei Yang

AI总结:

本文提出MM1.5多模态大语言模型系列,通过数据为中心的持续预训练与指令微调策略,在1B至30B规模上提升富文本图像理解、视觉定位、多图像推理及视频和UI理解能力。

摘要:

我们提出MM1.5,这是一个新的多模态大语言模型(MLLMs)系列,旨在增强富文本图像理解、视觉指代与定位以及多图像推理能力。基于MM1架构,MM1.5采用以数据为中心的模型训练方法,系统地探索了多样化数据混合在整个模型训练生命周期中的影响。这包括用于持续预训练的高质量OCR数据和合成字幕,以及用于监督微调的优化视觉指令微调数据混合。我们的模型参数规模从1B到30B不等,涵盖稠密模型和混合专家(MoE)变体,并证明即使在较小规模(1B和3B)下,精心的数据筛选和训练策略也能产生强大的性能。此外,我们引入了两个专用变体:MM1.5-Video,专为视频理解而设计,以及MM1.5-UI,专为移动UI理解而定制。通过广泛的实证研究和消融实验,我们提供了关于训练过程和决策的详细见解,这些见解为我们的最终设计提供了依据,并为未来MLLM开发研究提供了有价值的指导。

英文摘要:

We present MM1.5, a new family of multimodal large language models (MLLMs) designed to enhance capabilities in text-rich image understanding, visual referring and grounding, and multi-image reasoning. Building upon the MM1 architecture, MM1.5 adopts a data-centric approach to model training, systematically exploring the impact of diverse data mixtures across the entire model training lifecycle. This includes high-quality OCR data and synthetic captions for continual pre-training, as well as an optimized visual instruction-tuning data mixture for supervised fine-tuning. Our models range from 1B to 30B parameters, encompassing both dense and mixture-of-experts (MoE) variants, and demonstrate that careful data curation and training strategies can yield strong performance even at small scales (1B and 3B). Additionally, we introduce two specialized variants: MM1.5-Video, designed for video understanding, and MM1.5-UI, tailored for mobile UI understanding. Through extensive empirical studies and ablations, we provide detailed insights into the training processes and decisions that inform our final designs, offering valuable guidance for future research in MLLM development.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑