arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.21610cs.HC
收藏

WatchHand: 使智能手表实现连续手部姿态跟踪

WatchHand: Enabling Continuous Hand Pose Tracking On Off-the-Shelf Smartwatches

KAIST(韩国科学技术院);Cornell University(康奈尔大学)

Jiwan Kim, Chi-Jung Lee, Hohurn Jung, Tianhong Catherine Yu, Ruidong Zhang, Ian Oakley, Cheng Zhang

AI总结:

WatchHand是首个在商用智能手表上实现的连续3D手部姿态跟踪系统,利用内置扬声器和麦克风,通过声学信号处理实现20个指尖的3D姿态估计,有效降低智能手表手部跟踪的硬件门槛。

摘要:

在腕部可穿戴设备上进行手部姿态跟踪能够实现丰富的表达性交互,但目前在商用智能手表上仍不可用,因为先前的实现依赖于外部传感器或定制硬件,限制了其在现实中的应用。为此,我们提出了WatchHand,这是首个在商用智能手表上实现的连续3D手部姿态跟踪系统,仅使用其内置的扬声器和麦克风。WatchHand发出不可听的频率调制连续波,并捕捉其从手部的反射。这些声学信号通过深度学习模型处理,以估计20个指尖的3D姿态。我们评估了WatchHand在多种现实条件下的表现——多种智能手表型号、戴手方式、身体姿势、噪声条件、姿态变化协议等,并在跨会话测试中实现了平均每个关节位置误差为7.87毫米。虽然对于未见过的用户或手势性能会下降,但模型能够通过在少量数据上进行轻量级微调有效适应。总体而言,WatchHand通过消除额外硬件降低了智能手表基于手部跟踪的障碍,同时在数百万现有设备上实现了稳健且始终可用的交互。

英文摘要:

Tracking hand poses on wrist-wearables enables rich, expressive interactions, yet remains unavailable on commercial smartwatches, as prior implementations rely on external sensors or custom hardware, limiting their real-world applicability. To address this, we present WatchHand, the first continuous 3D hand pose tracking system implemented on off-the-shelf smartwatches using only their built-in speaker and microphone. WatchHand emits inaudible frequency-modulated continuous waves and captures their reflections from the hand. These acoustic signals are processed by a deep-learning model that estimates 3D hand poses for 20 finger joints. We evaluate WatchHand across diverse real-world conditions -- multiple smartwatch models, wearing-hands, body postures, noise conditions, pose-variation protocols -- and achieve a mean per-joint position error of 7.87 mm in cross-session tests with device remounting. Although performance drops for unseen users or gestures, the model adapts effectively with lightweight fine-tuning on small amounts of data. Overall, WatchHand lowers the barrier to smartwatch-based hand tracking by eliminating additional hardware while enabling robust, always-available interactions on millions of existing devices.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑