HRIBench：面向人机交互中实时人类感知的视觉语言模型基准测试
HRIBench: Benchmarking Vision-Language Models for Real-Time Human Perception in Human-Robot Interaction
AI总结：
本文提出HRIBench，一个评估人机交互中VLM实时人类感知能力的VQA基准，涵盖五个关键领域共1000个问题，评估11个模型发现现有VLM在核心感知和性能-延迟权衡上均不足，呼吁开发低延迟的小型VLM。
摘要：
实时人类感知对于有效的人机交互（HRI）至关重要。大型视觉语言模型（VLMs）提供了有前景的泛化感知能力，但通常存在高延迟问题，这会对用户体验产生负面影响，并限制了VLM在现实场景中的适用性。为了系统地研究HRI中人类感知的VLM能力及性能-延迟权衡，我们引入了HRIBench，这是一个视觉问答（VQA）基准，旨在跨对HRI至关重要的多种人类感知任务评估VLMs。HRIBench涵盖五个关键领域：（1）非言语线索理解，（2）言语指令理解，（3）人机物体关系理解，（4）社交导航，以及（5）人员识别。为构建HRIBench，我们从真实世界HRI环境中收集数据，为非言语线索理解策划问题，并为其余四个领域利用公开可用的数据集。我们为每个领域策划了200个VQA问题，HRIBench总计包含1000个问题。随后，我们在HRIBench上对最先进的闭源和开源VLMs（N=11）进行了全面评估。结果显示，尽管具备泛化能力，当前VLMs在HRI所需的核心感知能力上仍显不足。此外，实验中没有任何模型展现出适合实时部署的令人满意的性能-延迟权衡，这凸显了未来研究开发更小、低延迟且具备改进人类感知能力的VLMs的需求。HRIBench及结果见：https://github.com/interaction-lab/HRIBench。
英文摘要：
Real-time human perception is crucial for effective human-robot interaction (HRI). Large vision-language models (VLMs) offer promising generalizable perceptual capabilities but often suffer from high latency, which negatively impacts user experience and limits VLM applicability in real-world scenarios. To systematically study VLM capabilities in human perception for HRI and performance-latency trade-offs, we introduce HRIBench, a visual question-answering (VQA) benchmark designed to evaluate VLMs across a diverse set of human perceptual tasks critical for HRI. HRIBench covers five key domains: (1) non-verbal cue understanding, (2) verbal instruction understanding, (3) human-robot object relationship understanding, (4) social navigation, and (5) person identification. To construct HRIBench, we collected data from real-world HRI environments to curate questions for non-verbal cue understanding, and leveraged publicly available datasets for the remaining four domains. We curated 200 VQA questions for each domain, resulting in a total of 1000 questions for HRIBench. We then conducted a comprehensive evaluation of both state-of-the-art closed-source and open-source VLMs (N=11) on HRIBench. Our results show that, despite their generalizability, current VLMs still struggle with core perceptual capabilities essential for HRI. Moreover, none of the models within our experiments demonstrated a satisfactory performance-latency trade-off suitable for real-time deployment, underscoring the need for future research on developing smaller, low-latency VLMs with improved human perception capabilities. HRIBench and our results can be found in this Github repository: https://github.com/interaction-lab/HRIBench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
