VLsI：从大视觉语言模型到小模型的分层到交互言语化方法
VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models
AI总结：
针对视觉语言模型扩模带来的部署算力难题，研究提出含2B和7B规模的VLsI模型系列，通过带中间言语化器的逐层蒸馏对齐大模型推理，在10个基准上性能超GPT-4V且无需改动架构。
摘要：
近期，GPT-4V等闭源视觉语言模型（VLM）产出的高质量视觉指令微调样本数量激增，推动了不同规模开源VLM的快速发布。然而，通过扩大模型规模提升VLM性能会带来显著的计算挑战，尤其难以部署在移动平台、机器人等资源受限设备上。为解决这一问题，我们提出VLsI（分层到交互言语化），这是包含2B和7B两种规模的新型VLM系列，在保证精度的同时优先提升效率。VLsI采用独特的逐层蒸馏流程，引入中间“言语化器”将每一层的特征映射到自然语言空间，使小型VLM能够灵活对齐大型VLM的推理过程。该方法缓解了输出模仿中常见的训练不稳定问题，且超越了典型的最终层调优，实现了小型VLM与大型VLM逐层推理进程的对齐。我们在10个具有挑战性的视觉语言基准上验证了VLsI的性能，无需模型扩容、合并或架构改动，其性能较GPT-4V实现了显著提升（2B模型提升11.0%，7B模型提升17.4%）。
英文摘要：
The recent surge in high-quality visual instruction tuning samples from closed-source vision-language models (VLMs) such as GPT-4V has accelerated the release of open-source VLMs across various model sizes. However, scaling VLMs to improve performance using larger models brings significant computational challenges, especially for deployment on resource-constrained devices like mobile platforms and robots. To address this, we propose VLsI: Verbalized Layers-to-Interactions, a new VLM family in 2B and 7B model sizes, which prioritizes efficiency without compromising accuracy. VLsI leverages a unique, layer-wise distillation process, introducing intermediate "verbalizers" that map features from each layer to natural language space, allowing smaller VLMs to flexibly align with the reasoning processes of larger VLMs. This approach mitigates the training instability often encountered in output imitation and goes beyond typical final-layer tuning by aligning the small VLMs' layer-wise progression with that of the large ones. We validate VLsI across ten challenging vision-language benchmarks, achieving notable performance gains (11.0% for 2B and 17.4% for 7B) over GPT-4V without the need for model scaling, merging, or architectural changes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
