arXiv 2509.259162025-09-30cs.CVcs.CL

VLM-FO1：弥合视觉语言模型中高层推理与细粒度感知之间的差距

VLM-FO1: Bridging the Gap Between High-Level Reasoning and Fine-Grained Perception in VLMs

Om AI Research（Om AI研究机构）；Binjiang Institute of Zhejiang University（浙江大学滨江研究院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）

Peng Liu, Haozhan Shen, Chunxin Fang, Zhicheng Sun, Jiajia Liao, Tiancheng Zhao

备注：: 22 pages

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对视觉语言模型细粒度感知能力弱的问题，提出即插即用框架VLM-FO1，通过特征检索重构任务、双视觉编码器生成区域token，在多基准上取得SOTA且不损害通用视觉能力。

摘要：

视觉语言模型（VLM）擅长高层场景理解，但在需要精准定位的细粒度感知任务上表现不佳。这一缺陷源于根本性的不匹配：对于以语言为核心的架构而言，生成精确的数值坐标是一项极具挑战性的任务。本文提出VLM-FO1这一新型框架，通过将以物体为中心的感知从易出错的坐标生成问题重构为鲁棒的特征检索任务，突破了上述限制。\n我们的方法是即插即用模块，可与任意预训练VLM集成。它采用混合细粒度区域编码器（HFRE）——该编码器配备双视觉编码器——生成兼具丰富语义与空间细节的强区域表征token。随后，基于token的引用系统使大语言模型（LLM）能够顺畅地在这些特定视觉区域中进行推理并将语言落地。\n实验表明，VLM-FO1在多类基准测试中取得了当前最优性能，在物体定位、区域生成理解和视觉区域推理方面展现出卓越能力。关键在于，我们的两阶段训练策略确保了在获得这些感知能力提升的同时，不会损害基础模型的通用视觉理解能力。VLM-FO1为构建具备感知能力的VLM建立了有效且灵活的范式，弥合了高层推理与细粒度视觉定位之间的差距。

英文摘要：

Vision-Language Models (VLMs) excel at high-level scene understanding but falter on fine-grained perception tasks requiring precise localization. This failure stems from a fundamental mismatch, as generating exact numerical coordinates is a challenging task for language-centric architectures. In this paper, we introduce VLM-FO1, a novel framework that overcomes this limitation by reframing object-centric perception from a brittle coordinate generation problem into a robust feature retrieval task. Our method operates as a plug-and-play module that integrates with any pre-trained VLM. It leverages a Hybrid Fine-grained Region Encoder (HFRE), featuring a dual vision encoder, to generate powerful region tokens rich in both semantic and spatial detail. A token-based referencing system then enables the LLM to seamlessly reason about and ground language in these specific visual regions. Experiments show that VLM-FO1 achieves state-of-the-art performance across a diverse suite of benchmarks, demonstrating exceptional capabilities in object grounding, region generational understanding, and visual region reasoning. Crucially, our two-stage training strategy ensures that these perception gains are achieved without compromising the base model's general visual understanding capabilities. VLM-FO1 establishes an effective and flexible paradigm for building perception-aware VLMs, bridging the gap between high-level reasoning and fine-grained visual grounding.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。