GroundCount: 通过目标检测对视觉语言模型进行接地以缓解计数幻觉
GroundCount: Grounding Vision-Language Models with Object Detection for Mitigating Counting Hallucinations
AI总结：
GroundCount通过目标检测提供空间接地，缓解视觉语言模型计数幻觉，提升准确率并减少推理时间。
摘要：
视觉语言模型（VLMs）在计数任务中表现出持续的幻觉现象，其准确性显著低于其他视觉推理任务（不包括情感分析）。这种现象甚至在最先进的推理能力VLMs中仍然存在。相反，基于CNN的目标检测模型（ODMs）如YOLO在空间定位和实例计数方面表现出色，且计算开销极低。我们提出了GroundCount，一个通过ODMs为VLMs提供显式空间接地的框架，以缓解计数幻觉。在最佳情况下，我们的基于提示的增强策略在表现最佳的模型（Ovis2.5-2B）上实现了81.3%的计数准确率，比之前提高了6.6个百分点，同时通过消除幻觉驱动的推理循环，将推理时间减少了22%。我们进行了全面的消融研究，证明位置编码是一个关键组件，对更强的模型有益，但对较弱的模型有害。相比之下，置信度分数对大多数架构引入了噪声，其移除在五个评估的模型中有四个提高了性能。我们进一步评估了特征级融合架构，发现通过结构化提示进行显式符号接地优于隐式特征融合，尽管有复杂的跨注意力机制。我们的方法在四个评估的VLM架构中实现了持续的改进（6.2-7.5个百分点），其中有一个架构由于其迭代反思机制与结构化提示之间的不兼容性而性能下降。这些结果表明，计数失败源于基本的空间-语义整合限制，而不是架构特定的缺陷，同时突显了在增强策略中架构兼容性的重要性。
英文摘要：
Vision Language Models (VLMs) exhibit persistent hallucinations in counting tasks, with accuracy substantially lower than other visual reasoning tasks (excluding sentiment). This phenomenon persists even in state-of-the-art reasoning-capable VLMs. Conversely, CNN-based object detection models (ODMs) such as YOLO excel at spatial localization and instance counting with minimal computational overhead. We propose GroundCount, a framework that augments VLMs with explicit spatial grounding from ODMs to mitigate counting hallucinations. In the best case, our prompt-based augmentation strategy achieves 81.3% counting accuracy on the best-performing model (Ovis2.5-2B) - a 6.6pp improvement - while reducing inference time by 22% through elimination of hallucination-driven reasoning loops for stronger models. We conduct comprehensive ablation studies demonstrating that positional encoding is a critical component, being beneficial for stronger models but detrimental for weaker ones. Confidence scores, by contrast, introduce noise for most architectures and their removal improves performance in four of five evaluated models. We further evaluate feature-level fusion architectures, finding that explicit symbolic grounding via structured prompts outperforms implicit feature fusion despite sophisticated cross-attention mechanisms. Our approach yields consistent improvements across four of five evaluated VLM architectures (6.2--7.5pp), with one architecture exhibiting degraded performance due to incompatibility between its iterative reflection mechanisms and structured prompts. These results suggest that counting failures stem from fundamental spatial-semantic integration limitations rather than architecture-specific deficiencies, while highlighting the importance of architectural compatibility in augmentation strategies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
