HOI-Ref：第一人称视觉中的手-物交互指代
HOI-Ref: Hand-Object Interaction Referral in Egocentric Vision
AI总结：
本研究提出第一人称视觉手-物交互指代任务HOI-Ref，构建含390万问答对的HOI-QA数据集，训练出首个专用模型VLM4HOI，微调后手物指代和交互指代性能分别提升27.9%与26.7%。
摘要：
大视觉语言模型（VLM）目前已成为视觉问答、物体识别、空间指代等多项任务的事实标准最优方法。本研究针对第一人称图像提出HOI-Ref任务，旨在利用VLM理解手与物体之间的交互。为支撑HOI-Ref任务，我们构建了HOI-QA数据集，包含390万对问答对，用于VLM的训练与评估。HOI-QA涵盖与手部定位、物体定位相关的问题，核心还包含二者交互相关的问题（例如指代被手操作的物体）。我们基于该数据集训练了首个面向HOI-Ref的VLM，命名为VLM4HOI。实验结果表明，针对第三人称图像指代任务训练的VLM无法识别和指代第一人称图像中的手与物体；在我们的第一人称HOI-QA数据集上微调后，手与物体的指代性能提升27.9%，交互指代性能提升26.7%。
英文摘要：
Large Vision Language Models (VLMs) are now the de facto state-of-the-art for a number of tasks including visual question answering, recognising objects, and spatial referral. In this work, we propose the HOI-Ref task for egocentric images that aims to understand interactions between hands and objects using VLMs. To enable HOI-Ref, we curate the HOI-QA dataset that consists of 3.9M question-answer pairs for training and evaluating VLMs. HOI-QA includes questions relating to locating hands, objects, and critically their interactions (e.g. referring to the object being manipulated by the hand). We train the first VLM for HOI-Ref on this dataset and call it VLM4HOI. Our results demonstrate that VLMs trained for referral on third person images fail to recognise and refer hands and objects in egocentric images. When fine-tuned on our egocentric HOI-QA dataset, performance improves by 27.9% for referring hands and objects, and by 26.7% for referring interactions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
