Penguin-VL：探索基于大语言模型的视觉编码器的效率极限
Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders
AI总结：
本文提出Penguin-VL，通过基于大语言模型的视觉编码器替代传统对比预训练，实现更高效的多模态理解，在文档理解、视觉知识和多视角视频理解等任务中超越现有领先VLM。
摘要：
视觉语言模型（VLM）的发展主要依赖于扩大模型规模，这阻碍了在计算受限的移动和边缘设备（如智能手机和机器人）上的部署。在本工作中，我们探索紧凑（例如2B和8B）VLM的性能极限。我们挑战了现有做法，即最先进的VLM必须依赖通过大规模对比预训练（如CLIP/SigLIP）初始化的视觉编码器。我们发现了一个目标不匹配：对比学习，优化于辨别，强制粗粒度和类别级别的不变性，抑制了密集描述和复杂VLM推理所需的细粒度视觉线索。为了解决这个问题，我们提出了Penguin-VL，其视觉编码器从文本-only LLM初始化。我们的实验表明，Penguin-Encoder是传统对比预训练的更优替代品，解锁了更高程度的视觉保真度和数据效率。在各种图像和视频基准测试中，Penguin-VL在数学推理方面与领先VLM（如Qwen3-VL）表现相当，并在文档理解、视觉知识和多视角视频理解等任务中超越了它们。值得注意的是，这些增益是在轻量级架构上实现的，证明了改进的视觉表示而不是模型扩展是性能的主要驱动因素。我们的消融实验显示，Penguin-Encoder在对比预训练编码器上表现一致，保留了对密集感知和复杂推理至关重要的细粒度空间和时间线索。这使其成为计算高效VLM的强替代品，并在资源受限的环境中实现了高性能。代码：https://github.com/tencent-ailab/Penguin-VL
英文摘要：
Vision Language Model (VLM) development has largely relied on scaling model size, which hinders deployment on compute-constrained mobile and edge devices such as smartphones and robots. In this work, we explore the performance limits of compact (e.g., 2B and 8B) VLMs. We challenge the prevailing practice that state-of-the-art VLMs must rely on vision encoders initialized via massive contrastive pretraining (e.g., CLIP/SigLIP). We identify an objective mismatch: contrastive learning, optimized for discrimination, enforces coarse and category-level invariances that suppress fine-grained visual cues needed for dense captioning and complex VLM reasoning. To address this issue, we present Penguin-VL, whose vision encoder is initialized from a text-only LLM. Our experiments reveal that Penguin-Encoder serves as a superior alternative to traditional contrastive pretraining, unlocking a higher degree of visual fidelity and data efficiency for multimodal understanding. Across various image and video benchmarks, Penguin-VL achieves performance comparable to leading VLMs (e.g., Qwen3-VL) in mathematical reasoning and surpasses them in tasks such as document understanding, visual knowledge, and multi-perspective video understanding. Notably, these gains are achieved with a lightweight architecture, demonstrating that improved visual representation rather than model scaling is the primary driver of performance. Our ablations show that Penguin-Encoder consistently outperforms contrastive-pretrained encoders, preserving fine-grained spatial and temporal cues that are critical for dense perception and complex reasoning. This makes it a strong drop-in alternative for compute-efficient VLMs and enables high performance in resource-constrained settings. Code: https://github.com/tencent-ailab/Penguin-VL
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
