NanoVLM：我们能把视觉语言模型做到多小，同时仍保持输出连贯？
NanoVLMs: How small can we go and still make coherent Vision Language Models?
AI总结：
受幼儿视觉学习启发，本文构建 ShortDesc 与 LongDesc 数据集，证明可用简单架构训练比 SOTA 小型 VLM 小至多 10 倍且输出连贯的 NanoVLM。
摘要：
视觉语言模型（VLM），例如 GPT-4V 和 Llama 3.2 vision，因其能够在多模态任务中利用大语言模型（LLM）而获得了大量研究关注。然而，它们的潜力受到固有挑战的限制，包括专有限制、巨大的计算需求以及可访问性不足。较小的模型，如 GIT 和 BLIP，表现出明显的局限性：即使经过大量训练，也往往无法生成超过数个 token 的连贯且一致的文本。这凸显出一个关键问题：VLM 可以做到多小，同时仍能生成流畅且一致的文本？受 3 至 4 岁儿童出色学习过程的启发——他们在理解和交流时高度依赖视觉线索——我们引入了两个新颖数据集：ShortDesc（包含简洁的图像描述）和 LongDesc（包含更详细的图像描述）。这些数据集由图像-文本对组成，其中文本被限制为幼儿通常使用的简单词汇和句法，并由一个规模缩小的模型 GPT-4o 生成。利用这些数据集，我们证明可以训练出显著更小的 VLM：在保持架构简洁性的同时，其规模最多可比最先进（SOTA）的小型 VLM 小 10 倍。为评估输出，我们利用 GPT-4o 像批改学生所写故事一样，从创造性、意义性和一致性方面对文本进行评分，给出满分 10 分的分数。该方法通过容纳非结构化输出并对模型能力进行多维评估，解决了标准基准测试的局限性。我们的发现有助于面向资源受限环境开发轻量级、易于获取的多模态模型。
英文摘要：
Vision-Language Models (VLMs), such as GPT-4V and Llama 3.2 vision, have garnered significant research attention for their ability to leverage Large Language Models (LLMs) in multimodal tasks. However, their potential is constrained by inherent challenges, including proprietary restrictions, substantial computational demands, and limited accessibility. Smaller models, such as GIT and BLIP, exhibit marked limitations, often failing to generate coherent and consistent text beyond a few tokens, even with extensive training. This underscores a pivotal inquiry: how small can a VLM be and still produce fluent and consistent text? Drawing inspiration from the exceptional learning process of 3-4 year old children, who rely heavily on visual cues for understanding and communication, we introduce two novel datasets: ShortDesc (featuring concise image descriptions) and LongDesc (containing more detailed image descriptions). These datasets consist of image-text pairs where the text is restricted to the simple vocabulary and syntax typically used by young children, generated with a scaled-down model, GPT-4o. Using these datasets, we demonstrate that it is possible to train VLMs that are significantly smaller, up to 10 times smaller than state of the art(SOTA) small VLMs while maintaining architectural simplicity. To evaluate the outputs, we leverage GPT-4o to grade the text, as if stories written by students, on creativity, meaningfulness, and consistency, assigning scores out of 10. This method addresses limitations of standard benchmarks by accommodating unstructured outputs and providing a multidimensional evaluation of the model capabilities. Our findings contribute to the development of lightweight, accessible multimodal models for resource constrained environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
