THOR-Net:端到端基于Grformer的现实双手和具有自我监督的对象重构
THOR-Net: End-to-end Graformer-based Realistic Two Hands and Object Reconstruction with Self-supervision
摘要:
双手与物体交互的真实感重建是一个新的和具有挑战性的问题,是建立个性化虚拟和增强现实环境的必要条件。图卷积网络(GCN)通过将手的姿势和形状建模为图来允许保持手的姿势和形状的拓扑。在这项工作中,我们提出了THOR-Net,它结合了GCN,Transformer和自我监督的功能,可以从单个RGB图像中逼真地重建双手和物体。我们的网络包括两个阶段,即特征提取阶段和重建阶段。在特征提取阶段,使用Keypoint RCNN从单目RGB图像中提取2D姿势、特征图、热图和边界框。此后,该2D信息被建模为两个图,并被传递到重建阶段的两个分支。形状重建分支使用我们新颖的由粗到细的GraFormer形状网络来估计两只手和一个物体的网格。另一个分支使用GraFormer网络重建手部和物体的3D姿态。最后,一个自我监督的光度损失是用来直接回归的手的网格中的每个顶点的真实感纹理。我们的方法在HO-3D数据集(10.0mm)上获得了最先进的手部形状估计结果,超过了ArtiBoost(10.8mm)。它还在具有挑战性的双手和对象(H2O)数据集上的手部姿势估计中超过其他方法,左手姿势为5 mm,右手姿势为1 mm。
英文摘要:
Realistic reconstruction of two hands interacting with objects is a new and challenging problem that is essential for building personalized Virtual and Augmented Reality environments. Graph Convolutional networks (GCNs) allow for the preservation of the topologies of hands poses and shapes by modeling them as a graph. In this work, we propose the THOR-Net which combines the power of GCNs, Transformer, and self-supervision to realistically reconstruct two hands and an object from a single RGB image. Our network comprises two stages; namely the features extraction stage and the reconstruction stage. In the features extraction stage, a Keypoint RCNN is used to extract 2D poses, features maps, heatmaps, and bounding boxes from a monocular RGB image. Thereafter, this 2D information is modeled as two graphs and passed to the two branches of the reconstruction stage. The shape reconstruction branch estimates meshes of two hands and an object using our novel coarse-to-fine GraFormer shape network. The 3D poses of the hands and objects are reconstructed by the other branch using a GraFormer network. Finally, a self-supervised photometric loss is used to directly regress the realistic textured of each vertex in the hands' meshes. Our approach achieves State-of-the-art results in Hand shape estimation on the HO-3D dataset (10.0mm) exceeding ArtiBoost (10.8mm). It also surpasses other methods in hand pose estimation on the challenging two hands and object (H2O) dataset by 5mm on the left-hand pose and 1 mm on the right-hand pose.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。