User engagement is greatly enhanced by fully immersive multi-modal experiences that combine visual and auditory stimuli. Consequently, the next frontier in VR/AR technologies lies in immersive volumetric videos with complete scene capture, large 6-DoF interaction space, multi-modal feedback, and high resolution & frame-rate contents. To stimulate the reconstruction of immersive volumetric videos, we introduce ImViD, a multi-view, multi-modal dataset featuring complete space-oriented data capture and various indoor/outdoor scenarios. Our capture rig supports multi-view video-audio capture while on the move, a capability absent in existing datasets, significantly enhancing the completeness, flexibility, and efficiency of data capture.
The captured multi-view videos (with synchronized audios) are in 5K resolution at 60FPS, lasting from 1-5 minutes, and include rich foreground-background elements, and complex dynamics. We benchmark existing methods using our dataset and establish a base pipeline for constructing immersive volumetric videos from multi-view audiovisual inputs for 6-DoF multi-modal immersive VR experiences. The benchmark and the reconstruction and interaction results demonstrate the effectiveness of our dataset and baseline method, which we believe will stimulate future research on immersive volumetric video production.
Realizing Immersive Volumetric Video: A Multimodal Framework for 6-DoF VR Engagement
Tsinghua University（清华大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Migu Beijing Research Institute（咪咕北京研究院）；School of Architecture, Tsinghua University（清华大学建筑学院）；Department of Automation, Tsinghua University（清华大学自动化系）；BNRist, Tsinghua University（清华大学北京信息科学与技术国家研究中心）
We introduce HUMAN4D, a large and multimodal 4D dataset that contains a variety of human activities simultaneously captured by a professional marker-based MoCap, a volumetric capture and an audio recording system. By capturing 2 female and $2$ male professional actors performing various full-body movements and expressions, HUMAN4D provides a diverse set of motions and poses encountered as part of single- and multi-person daily, physical and social activities (jumping, dancing, etc.), along with multi-RGBD (mRGBD), volumetric and audio data. Despite the existence of multi-view color datasets ca…
ShanghaiTech University（上海科技大学）；Stereye Intelligent Technology Co.,Ltd.（视辰智能科技有限公司）；Huazhong University of Science and Technology（华中科技大学）
Some of the most exciting experiences that Metaverse promises to offer, for instance, live interactions with virtual characters in virtual environments, require real-time photo-realistic rendering. 3D reconstruction approaches to rendering, active or passive, still require extensive cleanup work to fix the meshes or point clouds. In this paper, we present a neural volumography technique called neural volumetric video or NeuVV to support immersive, interactive, and spatial-temporal rendering of volumetric video contents with photo-realism and in real-time. The core of NeuVV is to efficiently en…
West Virginia University（西弗吉尼亚大学）；Rockefeller Neuroscience Institute of West Virginia University（西弗吉尼亚大学洛克菲勒神经科学研究所）
The quest for deeper understanding of biological systems has driven the acquisition of increasingly larger multidimensional image datasets. Inspecting and manipulating data of this complexity is very challenging in traditional visualization systems. We developed syGlass, a software package capable of visualizing large scale volumetric data with inexpensive virtual reality head-mounted display technology. This allows leveraging stereoscopic vision to significantly improve perception of complex 3D structures, and provides immersive interaction with data directly in 3D. We accomplished this by de…
CPSL: Representing Volumetric Video via Content-Promoted Scene Layers
McGill University（麦吉尔大学）；University of Southern California（南加州大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Hybrid Foveated Path Tracing with Peripheral Gaussians for Immersive Anatomy
Technical University of Munich (TUM), Germany（慕尼黑技术大学）；TUM University Hospital, Orthopedics and Sports Orthopedics（TUM大学医院，骨科和运动骨科）；Human-Centered Computing and Extended Reality Lab (HEX)（以人为本计算与扩展现实实验室）；Munich Institute of Robotics and Machine Intelligence (MIRMI)（慕尼黑机器人与机器智能研究所）
ShanghaiTech University（上海科技大学）；Degene；Google（谷歌）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与影像工程研究中心）
4D reconstruction and rendering of human activities is critical for immersive VR/AR experience.Recent advances still fail to recover fine geometry and texture results with the level of detail present in the input images from sparse multi-view RGB cameras. In this paper, we propose NeuralHumanFVV, a real-time neural human performance capture and rendering system to generate both high-quality geometry and photo-realistic texture of human activities in arbitrary novel views. We propose a neural geometry generation scheme with a hierarchical sampling strategy for real-time implicit geometry infere…
Efficient Dynamic-NeRF Based Volumetric Video Coding with Rate Distortion Optimization
Institute of Image Communication and Network Engineering（图像通信与网络工程研究所）；Shanghai Jiao Tong University（上海交通大学）；College of Electronics and Information Engineering（电子信息工程学院）；Sichuan University（四川大学）
Multi-Layer Gaussian Splatting for Immersive Anatomy Visualization
Technical University of Munich（慕尼黑技术大学）；Clinic for Orthopedics and Sports Orthopedics（骨科与运动骨科诊所）；TUM University Hospital（慕尼黑技术大学医院）；Siemens Healthineers AG（西门子医疗影像有限公司）
With the emergence of neural radiance fields (NeRFs), view synthesis quality has reached an unprecedented level. Compared to traditional mesh-based assets, this volumetric representation is more powerful in expressing scene geometry but inevitably suffers from high rendering costs and can hardly be involved in further processes like editing, posing significant difficulties in combination with the existing graphics pipeline. In this paper, we present a hybrid volume-mesh representation, VMesh, which depicts an object with a textured mesh along with an auxiliary sparse volume. VMesh retains the …
CAGS: Color-Adaptive Volumetric Video Streaming with Dynamic 3D Gaussian Splatting
Simon Fraser University（西蒙弗雷泽大学）；Jiangxing Intelligence Inc.（江行智能有限公司）；McGill University（麦吉尔大学）；Nankai University（南开大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Fuzhou University（福州大学）；Southeast University（东南大学）
