arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.06438cs.CVcs.CLcs.LG
收藏

SOLO:用于可扩展视觉语言建模的单一Transformer

SOLO: A Single Transformer for Scalable Vision-Language Modeling

University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)

Yangyi Chen, Xingyao Wang, Hao Peng, Heng Ji

AI总结:

本文提出SOLO,一种单一Transformer架构的70亿参数视觉语言模型,通过从LLM初始化、顺序预训练和指令微调的开源训练方案,解决现有LVLM的可扩展性问题,性能媲美LLaVA-v1.5-7B,尤其在视觉数学推理上表现优异。

摘要:

我们提出了SOLO,一种用于可扩展视觉语言建模的单一Transformer。当前的大型视觉语言模型(LVLMs),如LLaVA,大多采用异构架构,将预训练的视觉编码器与大型语言模型(LLMs)连接起来,以促进视觉识别和复杂推理。尽管通过相对轻量级的训练实现了显著的性能,我们识别出四个主要的可扩展性限制:(1)视觉能力受限于预训练的视觉编码器,其规模通常比LLMs小一个数量级。(2)异构架构使既定硬件和软件基础设施的使用复杂化。(3)在此类架构上研究缩放定律必须考虑三个独立组件——视觉编码器、连接器和LLMs,这使分析复杂化。(4)使用现有视觉编码器通常需要遵循预定义的图像输入预处理规范,例如,将输入重塑为固定分辨率的方形图像,这给处理高分辨率图像或具有异常宽高比的图像以及在它们上进行训练带来了困难。统一的单一Transformer架构,如SOLO,有效解决了LVLMs中的这些可扩展性问题;然而,其在现代语境中的有限采用可能源于缺乏可靠的训练方案,该方案需平衡两种模态并确保十亿级模型的稳定训练。在本文中,我们介绍了首个用于开发SOLO的开源训练方案,SOLO是一个使用中等学术资源开发的开源70亿参数LVLM。该训练方案包括从LLMs初始化、在ImageNet和网络规模数据上进行顺序预训练,以及在我们策划的高质量数据集上进行指令微调。在广泛评估中,SOLO展示了与LLaVA-v1.5-7B相当的性能,尤其在视觉数学推理方面表现出色。

英文摘要:

We present SOLO, a single transformer for Scalable visiOn-Language mOdeling. Current large vision-language models (LVLMs) such as LLaVA mostly employ heterogeneous architectures that connect pre-trained visual encoders with large language models (LLMs) to facilitate visual recognition and complex reasoning. Although achieving remarkable performance with relatively lightweight training, we identify four primary scalability limitations: (1) The visual capacity is constrained by pre-trained visual encoders, which are typically an order of magnitude smaller than LLMs. (2) The heterogeneous architecture complicates the use of established hardware and software infrastructure. (3) Study of scaling laws on such architecture must consider three separate components - visual encoder, connector, and LLMs, which complicates the analysis. (4) The use of existing visual encoders typically requires following a pre-defined specification of image inputs pre-processing, for example, by reshaping inputs to fixed-resolution square images, which presents difficulties in processing and training on high-resolution images or those with unusual aspect ratio. A unified single Transformer architecture, like SOLO, effectively addresses these scalability concerns in LVLMs; however, its limited adoption in the modern context likely stems from the absence of reliable training recipes that balance both modalities and ensure stable training for billion-scale models. In this paper, we introduce the first open-source training recipe for developing SOLO, an open-source 7B LVLM using moderate academic resources. The training recipe involves initializing from LLMs, sequential pre-training on ImageNet and web-scale data, and instruction fine-tuning on our curated high-quality datasets. On extensive evaluation, SOLO demonstrates performance comparable to LLaVA-v1.5-7B, particularly excelling in visual mathematical reasoning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑