SmolVLM：重新定义小型高效多模态模型
SmolVLM: Redefining small and efficient multimodal models
AI总结：
SmolVLM系列通过优化架构、分词与数据筛选，以极小内存实现高效多模态推理，其中256M模型性能超越大300倍的Idefics-80B，2.2B模型媲美内存消耗翻倍的顶尖VLM。
摘要：
大型视觉语言模型（VLMs）性能卓越，但需要大量计算资源，这限制了它们在移动和边缘设备上的部署。较小的VLM通常沿用较大模型的设计选择，例如广泛的图像分词，导致GPU内存使用效率低下，并制约了设备端应用的实用性。我们推出了SmolVLM，这是一系列专为资源高效推理而设计的紧凑型多模态模型。我们系统地探索了针对低计算开销进行优化的架构配置、分词策略和数据筛选。通过这一过程，我们确定了关键的设计选择，这些选择在图像和视频任务上以最小的内存占用带来了显著的性能提升。我们最小的模型SmolVLM-256M在推理期间使用不到1GB的GPU内存，并且性能优于参数规模大300倍的Idefics-80B模型，尽管两者之间存在18个月的开发差距。我们最大的模型拥有2.2B参数，可与GPU内存消耗为其两倍的最先进VLM相媲美。SmolVLM模型不仅限于静态图像，还展示了稳健的视频理解能力。我们的结果强调，策略性的架构优化、激进而高效的分词以及精心筛选的训练数据能显著增强多模态性能，从而在显著更小的规模上促进实用且节能的部署。
英文摘要：
Large Vision-Language Models (VLMs) deliver exceptional performance but require significant computational resources, limiting their deployment on mobile and edge devices. Smaller VLMs typically mirror design choices of larger models, such as extensive image tokenization, leading to inefficient GPU memory usage and constrained practicality for on-device applications. We introduce SmolVLM, a series of compact multimodal models specifically engineered for resource-efficient inference. We systematically explore architectural configurations, tokenization strategies, and data curation optimized for low computational overhead. Through this, we identify key design choices that yield substantial performance gains on image and video tasks with minimal memory footprints. Our smallest model, SmolVLM-256M, uses less than 1GB GPU memory during inference and outperforms the 300-times larger Idefics-80B model, despite an 18-month development gap. Our largest model, at 2.2B parameters, rivals state-of-the-art VLMs consuming twice the GPU memory. SmolVLM models extend beyond static images, demonstrating robust video comprehension capabilities. Our results emphasize that strategic architectural optimizations, aggressive yet efficient tokenization, and carefully curated training data significantly enhance multimodal performance, facilitating practical, energy-efficient deployments at significantly smaller scales.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
