Mon3tr: 单目3D远程存在与预构建高斯人偶作为记忆化
Mon3tr: Monocular 3D Telepresence with Pre-built Gaussian Avatars as Amortization
AI总结：
Mon3tr通过单目3DGS技术实现远程存在，利用预构建的高斯人偶降低系统复杂性，实现实时高精度3D可视化与低带宽传输。
摘要：
沉浸式远程存在旨在通过使人类交互在AR/VR应用中变得更加真实，通过启用逼真全身体素描表示来增强远程协作。然而，现有系统依赖于硬件密集的多摄像头设置，并且需要高带宽的体积流媒体，限制了它们在移动设备上的实时性能。为了解决这些挑战，我们提出了Mon3tr，一种新的单目3D远程存在框架，首次将基于3D高斯散点（3DGS）的参数化人体建模整合到远程存在中。Mon3tr采用了一种记忆化计算策略，将过程分为一个一次性的离线多视图重建阶段，以构建用户特定的虚拟形象，以及在实时远程存在会话中的单目在线推断阶段。使用单个单目RGB摄像头实时捕获身体动作和面部表情，以驱动基于3DGS的参数化人体模型，显著降低了系统复杂性和成本。提取的运动和外观特征通过WebRTC的数据通道以<0.2 Mbps的速度传输，允许在网络波动下实现稳健的适应性。在接收端，例如Meta Quest 3，我们开发了一个轻量级的3DGS属性变形网络，动态地在预构建的虚拟形象上生成纠正的3DGS属性调整，合成逼真的运动和外观，以~60 FPS的速度。广泛的实验表明，我们的方法具有最先进的性能，实现了新的姿态下的PSNR>28 dB，端到端延迟~80 ms，以及与点云流媒体相比的>1000倍带宽减少，同时支持从单目输入中实时操作，适用于多种场景。我们的演示可在https://mon3tr3d.github.io上找到。
英文摘要：
Immersive telepresence aims to transform human interaction in AR/VR applications by enabling lifelike full-body holographic representations for enhanced remote collaboration. However, existing systems rely on hardware-intensive multi-camera setups and demand high bandwidth for volumetric streaming, limiting their real-time performance on mobile devices. To overcome these challenges, we propose Mon3tr, a novel Monocular 3D telepresence framework that integrates 3D Gaussian splatting (3DGS) based parametric human modeling into telepresence for the first time. Mon3tr adopts an amortized computation strategy, dividing the process into a one-time offline multi-view reconstruction phase to build a user-specific avatar and a monocular online inference phase during live telepresence sessions. A single monocular RGB camera is used to capture body motions and facial expressions in real time to drive the 3DGS-based parametric human model, significantly reducing system complexity and cost. The extracted motion and appearance features are transmitted at < 0.2 Mbps over WebRTC's data channel, allowing robust adaptation to network fluctuations. On the receiver side, e.g., Meta Quest 3, we develop a lightweight 3DGS attribute deformation network to dynamically generate corrective 3DGS attribute adjustments on the pre-built avatar, synthesizing photorealistic motion and appearance at ~ 60 FPS. Extensive experiments demonstrate the state-of-the-art performance of our method, achieving a PSNR of > 28 dB for novel poses, an end-to-end latency of ~ 80 ms, and > 1000x bandwidth reduction compared to point-cloud streaming, while supporting real-time operation from monocular inputs across diverse scenarios. Our demos can be found at https://mon3tr3d.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
