实现沉浸式体积分量视频：一种多模态框架用于6自由度VR互动
Realizing Immersive Volumetric Video: A Multimodal Framework for 6-DoF VR Engagement
AI总结：
本文提出了一种多模态框架，用于构建沉浸式体积分量视频，通过多视角多模态数据集和动态光场重建框架，实现高分辨率、高帧率的动态内容，支持6自由度VR交互。
摘要：
完全沉浸式体验需要紧密整合6自由度视觉和听觉交互，这对于虚拟和增强现实至关重要。虽然可以通过计算机生成内容实现此类体验，但直接从现实世界捕获的视频中构建它们仍鲜为人知。我们引入了沉浸式体积分量视频，一种新的体积分量媒体格式，旨在提供大6自由度交互空间、视听反馈和高分辨率、高帧率动态内容。为了支持IVV构建，我们提出了ImViD，一个基于空间导向捕获哲学的多视角、多模态数据集。我们的自定义捕获装置在运动过程中同步多视角视频-音频采集，从而高效捕获复杂室内和室外场景，具有丰富的前景-背景交互和挑战性动态。数据集提供了5K分辨率视频，每秒60帧，持续时间1-5分钟，比现有基准更具丰富的空间、时间和多模态覆盖。利用此数据集，我们开发了一个动态光场重建框架，基于高斯基的时空表示，结合流引导稀疏初始化、联合相机时间校准和多术语时空监督，以稳健和准确地建模复杂运动。我们进一步提出，据我们所知，首次从此类多视角音频视频数据中重建声音场的方法。这些组件共同形成一个统一的沉浸式体积分量视频生产管道。广泛的基准测试和沉浸式VR实验表明，我们的管道生成高质量、时间稳定的视听体积分量内容，具有大6自由度交互空间。这项工作为沉浸式体积分量视频提供了基础定义和实用构建方法。
英文摘要：
Fully immersive experiences that tightly integrate 6-DoF visual and auditory interaction are essential for virtual and augmented reality. While such experiences can be achieved through computer-generated content, constructing them directly from real-world captured videos remains largely unexplored. We introduce Immersive Volumetric Videos, a new volumetric media format designed to provide large 6-DoF interaction spaces, audiovisual feedback, and high-resolution, high-frame-rate dynamic content. To support IVV construction, we present ImViD, a multi-view, multi-modal dataset built upon a space-oriented capture philosophy. Our custom capture rig enables synchronized multi-view video-audio acquisition during motion, facilitating efficient capture of complex indoor and outdoor scenes with rich foreground--background interactions and challenging dynamics. The dataset provides 5K-resolution videos at 60 FPS with durations of 1-5 minutes, offering richer spatial, temporal, and multimodal coverage than existing benchmarks. Leveraging this dataset, we develop a dynamic light field reconstruction framework built upon a Gaussian-based spatio-temporal representation, incorporating flow-guided sparse initialization, joint camera temporal calibration, and multi-term spatio-temporal supervision for robust and accurate modeling of complex motion. We further propose, to our knowledge, the first method for sound field reconstruction from such multi-view audiovisual data. Together, these components form a unified pipeline for immersive volumetric video production. Extensive benchmarks and immersive VR experiments demonstrate that our pipeline generates high-quality, temporally stable audiovisual volumetric content with large 6-DoF interaction spaces. This work provides both a foundational definition and a practical construction methodology for immersive volumetric videos.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
