arXiv 2605.122972026-05-12cs.CVcs.ROeess.IV

EgoEV-HandPose：基于立体事件相机的视角3D手姿估计与手势识别

EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Cameras

National Research Center for Optical Instrumentation, Zhejiang University（浙江大学光学仪器国家研究中心）；School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（湖南大学人工智能与机器人学院和机器人视觉感知与控制技术国家工程研究中心）；Ant Group Company Ltd.（蚂蚁集团有限公司）

Luming Wang, Hao Shi, Jiajun Zhai, Kailun Yang, Kaiwei Wang

备注：: Extended version of SMC 2025 paper arXiv:2503.12419. The established dataset and source code will be publicly released at https://github.com/ZJUWang01/EgoEV-HandPose

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出EgoEV-HandPose框架，通过立体事件流实现3D双臂姿态估计和手势识别，引入KeypointBEV模块和EgoEVHands数据集，取得优异性能。

摘要：

视角3D手姿估计和手势识别对于沉浸式增强/虚拟现实、人机交互和机器人至关重要。然而，传统帧基相机存在运动模糊和动态范围有限的问题，现有事件基方法受限于自身运动干扰、单目深度模糊和缺乏大规模真实立体数据集。为克服这些限制，我们提出了EgoEV-HandPose，一种端到端框架，用于从立体事件流中联合估计3D双臂姿态和识别手势。核心方法是KeypointBEV，一个灵活的立体融合模块，将特征提升到规范的鸟瞰空间，并采用迭代投影引导的精修循环逐步解决深度不确定性并强制运动学一致性。此外，我们引入EgoEVHands，首个大规模真实立体事件相机数据集，包含5,419个标注序列，涵盖38种手势类别下的密集3D/2D关键点。大量实验表明，EgoEV-HandPose在MPJPE为30.54mm和86.87%的Top-1手势识别准确率上取得最佳性能，显著优于RGB基立体和先前事件相机方法，特别是在低光和双臂遮挡场景中，从而为事件基视角感知设立了新基准。建立的数据集和源代码将在https://github.com/ZJUWang01/EgoEV-HandPose公开发布。

英文摘要：

Egocentric 3D hand pose estimation and gesture recognition are essential for immersive augmented/virtual reality, human-computer interaction, and robotics. However, conventional frame-based cameras suffer from motion blur and limited dynamic range, while existing event-based methods are hindered by ego-motion interference, monocular depth ambiguity, and the lack of large-scale real-world stereo datasets. To overcome these limitations, we propose EgoEV-HandPose, an end-to-end framework for joint 3D bimanual pose estimation and gesture recognition from stereo event streams. Central to our approach is KeypointBEV, a flexible stereo fusion module that lifts features into a canonical bird's-eye-view space and employs an iterative reprojection-guided refinement loop to progressively resolve depth uncertainty and enforce kinematic consistency. In addition, we introduce EgoEVHands, the first large-scale real-world stereo event-camera dataset for egocentric hand perception, containing 5,419 annotated sequences with dense 3D/2D keypoints across 38 gesture classes under varying illumination. Extensive experiments demonstrate that EgoEV-HandPose achieves state-of-the-art performance with an MPJPE of 30.54mm and 86.87% Top-1 gesture recognition accuracy, significantly outperforming RGB-based stereo and prior event-camera methods, particularly in low-light and bimanual occlusion scenarios, thereby setting a new benchmark for event-based egocentric perception. The established dataset and source code will be publicly released at https://github.com/ZJUWang01/EgoEV-HandPose.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。