小型语言模型遇上增强视觉词汇表
Small Language Model Meets with Reinforced Vision Vocabulary
AI总结:
本文提出小型视觉语言模型Vary-toy,以Qwen-1.8B为语言基座,并用目标检测正样本改进视觉词汇表,使其可在GTX1080ti上训练部署,同时在多个基准上取得有效性能。
摘要:
在2023年,使用大型视觉语言模型(LVLMs)是AI社区的流行趋势。然而,主流LVLMs相对较多的参数量(超过7B)使其难以在消费级GPU上训练和部署,令许多资源有限的研究者望而却步。想象一下,在一块老旧的GTX1080ti(我们唯一的游戏显卡)上体验当前LVLMs的全部功能会有多酷。因此,我们在本报告中提出了Vary-toy,这是一个小型Vary,并以Qwen-1.8B作为基础的“大”语言模型。在Vary-toy中,我们引入了改进的视觉词汇表,使模型不仅具备Vary的全部功能,还能获得更强的通用性。具体而言,在生成视觉词汇表的过程中,我们用目标检测驱动的正样本数据替换自然图像的负样本,从而更充分地利用词汇网络的容量,并使其能够高效编码与自然物体对应的视觉信息。实验方面,Vary-toy在DocVQA上取得65.6%的ANLS,在ChartQA上取得59.1%的准确率,在RefCOCO上取得88.1%的准确率,并在MMVet上取得29%。代码将在主页公开发布。
英文摘要:
Playing Large Vision Language Models (LVLMs) in 2023 is trendy among the AI community. However, the relatively large number of parameters (more than 7B) of popular LVLMs makes it difficult to train and deploy on consumer GPUs, discouraging many researchers with limited resources. Imagine how cool it would be to experience all the features of current LVLMs on an old GTX1080ti (our only game card). Accordingly, we present Vary-toy in this report, a small-size Vary along with Qwen-1.8B as the base ``large'' language model. In Vary-toy, we introduce an improved vision vocabulary, allowing the model to not only possess all features of Vary but also gather more generality. Specifically, we replace negative samples of natural images with positive sample data driven by object detection in the procedure of generating vision vocabulary, more sufficiently utilizing the capacity of the vocabulary network and enabling it to efficiently encode visual information corresponding to natural objects. For experiments, Vary-toy can achieve 65.6% ANLS on DocVQA, 59.1% accuracy on ChartQA, 88.1% accuracy on RefCOCO, and 29% on MMVet. The code will be publicly available on the homepage.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。