EgoChoir：从第一人称视角捕捉3D人-物交互区域
EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Views
AI总结：
针对第一人称视角下人-物交互区域捕捉的难题，提出EgoChoir框架，通过协调视觉外观、头部运动和3D物体结构，联合推断3D人体接触与物体可供性，并利用梯度调制适应多种场景，实验证明其有效且优越。
摘要：
理解第一人称视角下的人-物交互（HOI）是以人为中心的感知的基础，有助于促进AR/VR和具身AI等应用。对于第一人称HOI，除了感知语义（例如“什么”交互正在发生），捕捉交互在3D空间中具体表现在“哪里”也至关重要，这连接了感知与操作。现有方法主要利用HOI的观察从非第一人称视角捕捉交互区域。然而，第一人称视角下交互方的不完整观察在视觉观察与交互内容之间引入了歧义，损害了其有效性。从第一人称视角来看，人类整合视觉皮层、小脑和大脑来内化他们的意图和物体的交互概念，即使在交互区域不可见时也能预先形成交互并做出行为。鉴于此，我们提出协调视觉外观、头部运动和3D物体来挖掘物体交互概念和主体意图，从第一人称视频中联合推断3D人体接触和物体可供性。为了实现这一点，我们提出了EgoChoir，它将物体结构与外观和头部运动中固有的交互上下文联系起来以揭示物体可供性，进一步利用它来建模人体接触。此外，采用梯度调制来采用适当的线索，以在各种第一人称场景中捕捉交互区域。此外，为从Ego-Exo4D和GIMO收集的第一人称视频标注了3D接触和可供性以支持该任务。对它们的广泛实验证明了EgoChoir的有效性和优越性。
英文摘要：
Understanding egocentric human-object interaction (HOI) is a fundamental aspect of human-centric perception, facilitating applications like AR/VR and embodied AI. For the egocentric HOI, in addition to perceiving semantics e.g., ''what'' interaction is occurring, capturing ''where'' the interaction specifically manifests in 3D space is also crucial, which links the perception and operation. Existing methods primarily leverage observations of HOI to capture interaction regions from an exocentric view. However, incomplete observations of interacting parties in the egocentric view introduce ambiguity between visual observations and interaction contents, impairing their efficacy. From the egocentric view, humans integrate the visual cortex, cerebellum, and brain to internalize their intentions and interaction concepts of objects, allowing for the pre-formulation of interactions and making behaviors even when interaction regions are out of sight. In light of this, we propose harmonizing the visual appearance, head motion, and 3D object to excavate the object interaction concept and subject intention, jointly inferring 3D human contact and object affordance from egocentric videos. To achieve this, we present EgoChoir, which links object structures with interaction contexts inherent in appearance and head motion to reveal object affordance, further utilizing it to model human contact. Additionally, a gradient modulation is employed to adopt appropriate clues for capturing interaction regions across various egocentric scenarios. Moreover, 3D contact and affordance are annotated for egocentric videos collected from Ego-Exo4D and GIMO to support the task. Extensive experiments on them demonstrate the effectiveness and superiority of EgoChoir.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
