arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2201.09548cs.CVcs.AI
收藏

基于自监督学习的视频一致性三维Hand重建

Consistent 3D Hand Reconstruction in Video via self-supervised Learning

Wuhan University(武汉大学);Technical University of Munich(慕尼黑工业大学);Tencent AI Lab(腾讯人工智能实验室);University at Buffalo(布法罗大学)

Zhigang Tu, Zhisheng Huang, Yujin Chen, Di Kang, Linchao Bao, Bisheng Yang, Junsong Yuan

摘要:

我们提出了一种方法重建准确和一致的3D手从单目视频。我们观察到,检测到的2D手部关键点和图像纹理提供了关于3D手部的几何形状和纹理的重要线索,这可以减少甚至消除对3D手部标注的要求。因此,我们提出了${\rm {S}^{2}HAND}$,一个自我监督的3D手部重建模型,它可以通过监督易于访问的2D检测到的关键点,从单个RGB输入联合估计姿势,形状,纹理和相机视点。我们利用未标记的视频数据中包含的连续手部运动信息,提出了${\rm {S}^{2}HAND(V)}$,它使用一组共享的权重${\rm {S}^{2}HAND}$来处理每一帧,并利用额外的运动,纹理和形状一致性约束,以促进更准确的手部姿势和更一致的形状和纹理。在基准数据集上的实验表明,与目前的全监督方法相比,该方法在单帧输入下具有相当的手部重建性能,并且在使用视频训练数据时显著提高了重建精度和一致性.

英文摘要:

We present a method for reconstructing accurate and consistent 3D hands from a monocular video. We observe that detected 2D hand keypoints and the image texture provide important cues about the geometry and texture of the 3D hand, which can reduce or even eliminate the requirement on 3D hand annotation. Thus we propose ${\rm {S}^{2}HAND}$, a self-supervised 3D hand reconstruction model, that can jointly estimate pose, shape, texture, and the camera viewpoint from a single RGB input through the supervision of easily accessible 2D detected keypoints. We leverage the continuous hand motion information contained in the unlabeled video data and propose ${\rm {S}^{2}HAND(V)}$, which uses a set of weights shared ${\rm {S}^{2}HAND}$ to process each frame and exploits additional motion, texture, and shape consistency constrains to promote more accurate hand poses and more consistent shapes and textures. Experiments on benchmark datasets demonstrate that our self-supervised approach produces comparable hand reconstruction performance compared with the recent full-supervised methods in single-frame as input setup, and notably improves the reconstruction accuracy and consistency when using video training data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑