BIGS：基于单目视频的3D高斯溅射实现与类别无关的双手交互重建
BIGS: Bimanual Category-agnostic Interaction Reconstruction from Monocular Videos via 3D Gaussian Splatting
AI总结：
针对单目RGB视频中两只手与未知物体交互的重建难题，提出BIGS方法，通过3D高斯溅射结合扩散模型先验与MANO手部先验，解决遮挡问题并实现手-物3D对齐，在多项指标上达到SOTA精度。
摘要：
重建手-物交互（HOI）的3D模型是一个具有众多应用的基础问题。尽管近年来取得了进展，但目前尚无针对单目RGB视频中与类别无关的双手交互重建的综合 pipeline——在此场景中，两只手与一个未知物体相互作用。以往工作仅处理有限的手-物交互情况，即物体模板预先已知或仅一只手参与交互。双手交互重建因两只手与物体间的复杂交互存在严重遮挡。为解决此问题，我们首先提出BIGS（双手交互3D高斯溅射），一种从单目视频重建手和未知物体3D高斯模型的方法。为稳健获取物体高斯模型以避开严重遮挡，我们利用预训练扩散模型的先验知识，结合分数蒸馏采样（SDS）损失，重建未观测到的物体部分。对于手的高斯模型，考虑到有限视角，我们利用手部模型（即MANO）的3D先验，并为两只手共享单个高斯模型以有效积累手部3D信息。为进一步考虑手与物体间的3D对齐，我们在高斯优化过程中加入交互主体优化步骤。我们的方法在两个具有挑战性的数据集上，在3D手部姿态估计（MPJPE）、3D物体重建（CDh、CDo、F10）和渲染质量（PSNR、SSIM、LPIPS）方面均达到了最先进的精度。
英文摘要：
Reconstructing 3Ds of hand-object interaction (HOI) is a fundamental problem that can find numerous applications. Despite recent advances, there is no comprehensive pipeline yet for bimanual class-agnostic interaction reconstruction from a monocular RGB video, where two hands and an unknown object are interacting with each other. Previous works tackled the limited hand-object interaction case, where object templates are pre-known or only one hand is involved in the interaction. The bimanual interaction reconstruction exhibits severe occlusions introduced by complex interactions between two hands and an object. To solve this, we first introduce BIGS (Bimanual Interaction 3D Gaussian Splatting), a method that reconstructs 3D Gaussians of hands and an unknown object from a monocular video. To robustly obtain object Gaussians avoiding severe occlusions, we leverage prior knowledge of pre-trained diffusion model with score distillation sampling (SDS) loss, to reconstruct unseen object parts. For hand Gaussians, we exploit the 3D priors of hand model (i.e., MANO) and share a single Gaussian for two hands to effectively accumulate hand 3D information, given limited views. To further consider the 3D alignment between hands and objects, we include the interacting-subjects optimization step during Gaussian optimization. Our method achieves the state-of-the-art accuracy on two challenging datasets, in terms of 3D hand pose estimation (MPJPE), 3D object reconstruction (CDh, CDo, F10), and rendering quality (PSNR, SSIM, LPIPS), respectively.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
