arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.26362cs.CV
收藏

HandVQA: 评估视觉-语言模型中手部精细空间推理的诊断与改进

HandVQA: Diagnosing and Improving Fine-Grained Spatial Reasoning about Hands in Vision-Language Models

UNIST(蔚山科学技术院);University of Aberdeen(阿伯丁大学);University College London(伦敦大学学院);Fogsphere (Redev.AI Ltd), UK(Fogsphere (Redev.AI Ltd),英国)

MD Khalequzzaman Chowdhury Sayem, Mubarrat Tajoar Chowdhury, Yihalem Yimolal Tiruneh, Muneeb A. Khan, Muhammad Salman Ali, Binod Bhattarai, Seungryul Baek

AI总结:

HandVQA通过视觉问答评估VLM对手部解剖的精细空间理解,发现现有模型在手部关节空间关系推理上的系统性缺陷,并通过3D数据训练提升模型在手部姿态识别和手-物交互任务中的性能。

摘要:

理解人类手部的精细运动在高风险领域如机器人辅助手术、芯片制造和AR/VR人机交互中至关重要。尽管在通用视觉-语言基准上表现接近人类,当前视觉-语言模型(VLMs)在处理复杂和精细的手部姿势时仍存在困难。我们引入HandVQA,一个大规模诊断基准,旨在通过视觉问答评估VLM对手部解剖细节的理解。基于高质量的3D手部数据集(FreiHAND、InterHand2.6M、FPHA),该基准包含超过160万个多选问题,探究手关节之间的空间关系,如角度、距离和相对位置。我们评估了几种最先进的VLMs(LLaVA、DeepSeek和Qwen-VL)在基础和微调设置下的表现,使用轻量级微调方法LoRA。我们的发现揭示了当前模型的系统性限制,包括虚构的手指部分、错误的几何解释和较差的泛化能力。HandVQA不仅暴露了这些关键推理缺口,还提供了一条改进的验证路径。我们证明了从该基准学习到的3D空间知识在零样本设置下转移,显著提高了模型在新型下游任务如手部姿态识别(+10.33%)和手-物交互(+2.63%)中的准确性。

英文摘要:

Understanding the fine-grained articulation of human hands is critical in high-stakes settings such as robot-assisted surgery, chip manufacturing, and AR/VR-based human-AI interaction. Despite achieving near-human performance on general vision-language benchmarks, current vision-language models (VLMs) struggle with fine-grained spatial reasoning, especially in interpreting complex and articulated hand poses. We introduce HandVQA, a large-scale diagnostic benchmark designed to evaluate VLMs' understanding of detailed hand anatomy through visual question answering. Built upon high-quality 3D hand datasets (FreiHAND, InterHand2.6M, FPHA), our benchmark includes over 1.6M controlled multiple-choice questions that probe spatial relationships between hand joints, such as angles, distances, and relative positions. We evaluate several state-of-the-art VLMs (LLaVA, DeepSeek and Qwen-VL) in both base and fine-tuned settings, using lightweight fine-tuning via LoRA. Our findings reveal systematic limitations in current models, including hallucinated finger parts, incorrect geometric interpretations, and poor generalization. HandVQA not only exposes these critical reasoning gaps but provides a validated path to improvement. We demonstrate that the 3D-grounded spatial knowledge learned from our benchmark transfers in a zero-shot setting, significantly improving accuracy of model on novel downstream tasks like hand gesture recognition (+10.33%) and hand-object interaction (+2.63%).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑