金点狙击手：VLM中的自引导视觉推理用于细粒度动作理解
Gold Points Sniper: Self-guided Visual Reasoning in VLM for Fine-grained Action Understanding
AI总结：
提出金点狙击手框架，通过金点提取器、选择性苏格拉底提问器和语义蕴含评估器三个模块，增强轻量级VLM的细粒度人类动作理解能力，在CAP基准上达到接近GPT-4o的性能且事实准确性更优。
摘要：
在日常环境中运行的机器人必须理解细粒度的人类动作、意图和上下文线索，这些线索来自人类仅占据小区域的广阔视角，而当前系统无法满足这一能力。虽然开放词汇动作识别方法仍局限于分配预定义标签，视觉语言模型（VLM）在其输出中面临信息丰富性和事实保真度之间的固有权衡，这两种方法都无法实现可靠人机交互所需的深层语义解释。我们提出金点狙击手（GPS），一种新颖框架，赋予轻量级VLM自引导多模态推理能力，用于细粒度人类动作理解。我们的方法包括三个关键模块：金点提取器训练VLM识别关键动作相关细节，选择性苏格拉底提问器通过选择性自我提问验证和细化这些细节，语义蕴含评估器使用语义蕴含分类定量评估事实一致性。在我们基于CAP基准策划的指令微调数据集上的大量实验表明，GPS增强的轻量级VLM实现了显著的性能提升，某些模型达到与专有GPT-4o相当的性能，同时保持更优的事实准确性。我们的工作为家庭机器人中的细粒度动作理解建立了可靠基础，使机器人能够通过信息密集但事实准确的描述安全地解释人类行为。源代码、训练配置、注释提示和数据集详情发布在此https URL。
英文摘要：
Robots operating in everyday environments must understand fine-grained human actions, intentions, and contextual cues from broad views where people occupy only small regions, a capability unmet by current systems. While open-vocabulary action recognition methods remain limited to assigning predefined labels, and vision-language models (VLMs) face an inherent trade-off between informational richness and factual fidelity in their outputs, neither approach achieves the deep semantic interpretation required for reliable human-robot interaction. We propose Gold Points Sniper (GPS), a novel framework that empowers lightweight VLMs with self-guided multimodal reasoning capabilities for fine-grained human action understanding. Our approach comprises three key modules: Gold Points Extractor trains VLMs to identify critical action-relevant details, Selective Socratic Questioner validates and refines these details through selective self-questioning, and Semantic Entailment Evaluator quantitatively assesses factual consistency using semantic entailment classification. Extensive experiments on our curated instruction-tuning dataset based on the CAP benchmark demonstrate that GPS-enhanced lightweight VLMs achieve substantial performance improvements, with some models reaching performance comparable to proprietary GPT-4o while maintaining superior factual accuracy. Our work establishes a reliable foundation for fine-grained action understanding in domestic robotics, enabling robots to safely interpret human behavior through information-dense yet factually grounded descriptions. Source code, training configurations, annotation prompts, and dataset details are released at https://github.com/Haodi-Liu/GPS-Gold-Point-Sniper.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
