Online free-viewpoint video (FVV) streaming is a challenging problem, which is relatively under-explored. It requires incremental on-the-fly updates to a volumetric representation, fast training and rendering to satisfy real-time constraints and a small memory footprint for efficient transmission. If achieved, it can enhance user experience by enabling novel applications, e.g., 3D video conferencing and live volumetric video broadcast, among others. In this work, we propose a novel framework for QUantized and Efficient ENcoding (QUEEN) for streaming FVV using 3D Gaussian Splatting (3D-GS). QUEEN directly learns Gaussian attribute residuals between consecutive frames at each time-step without imposing any structural constraints on them, allowing for high quality reconstruction and generalizability. To efficiently store the residuals, we further propose a quantization-sparsity framework, which contains a learned latent-decoder for effectively quantizing attribute residuals other than Gaussian positions and a learned gating module to sparsify position residuals. We propose to use the Gaussian viewspace gradient difference vector as a signal to separate the static and dynamic content of the scene. It acts as a guide for effective sparsity learning and speeds up training. On diverse FVV benchmarks, QUEEN outperforms the state-of-the-art online FVV methods on all metrics. Notably, for several highly dynamic scenes, it reduces the model size to just 0.7 MB per frame while training in under 5 sec and rendering at 350 FPS. Project website is at https://research.nvidia.com/labs/amri/projects/queen
Motion Matters: Compact Gaussian Streaming for Free-Viewpoint Video Reconstruction
College of Applied Technology, Shenzhen University（深圳大学应用技术学院）；College of Big Data and Internet, Shenzhen Technology University（深圳科技大学大数据与互联网学院）；College of Electronics and Information Engineering, Shenzhen University（深圳大学电子与信息工程学院）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）；Pengcheng Laboratory, Shenzhen（深圳鹏城实验室）；School of Electronic and Computer Engineering, Peking University（北京大学电子与计算机工程学院）
PAS3R: Pose-Adaptive Streaming 3D Reconstruction for Long Video Sequences
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi'an Jiaotong University, China（人机混合增强智能国家重点实验室，人工智能与机器人研究院，西安交通大学，中国）；University of East Anglia, Norwich, NR47TJ, United Kingdom（东安格利亚大学，诺里奇，英国）
LoD-Structured 3D Gaussian Splatting for Streaming Video Reconstruction
School of Computing and Data Science, The University of Hong Kong（计算与数据科学学院，香港大学）；Futurewei Technologies Inc（未来科技公司）；School of Computer and Information Sciences, The University of Newcastle（计算机与信息科学学院，新castle大学）
Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；McGill University（麦吉尔大学）；Intel Labs China, Peking University（北京大学英特尔中国实验室）
Semantic understanding of 3D point clouds is important for various robotics applications. Given that point-wise semantic annotation is expensive, in this paper, we address the challenge of learning models with extremely sparse labels. The core problem is how to leverage numerous unlabeled points. To this end, we propose a self-supervised 3D representation learning framework named viewpoint bottleneck. It optimizes a mutual-information based objective, which is applied on point clouds under different viewpoints. A principled analysis shows that viewpoint bottleneck leads to an elegant surrogate…
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China（脑启发式智能感知与认知联合实验室，中国科学技术大学）；University of Missouri, Kansas City（密苏里大学，堪萨斯城）
Recent works in spatiotemporal radiance fields can produce photorealistic free-viewpoint videos. However, they are inherently unsuitable for interactive streaming scenarios (e.g. video conferencing, telepresence) because have an inevitable lag even if the training is instantaneous. This is because these approaches consume videos and thus have to buffer chunks of frames (often seconds) before processing. In this work, we take a step towards interactive streaming via a frame-by-frame approach naturally free of lag. Conventional wisdom believes that per-frame NeRFs are impractical due to prohibit…
Scale-GS: Efficient Scalable Gaussian Splatting via Redundancy-filtering Training on Streaming Content
School of Computer Science and Technology, Dalian University of Technology（大连理工大学计算机科学与技术学院）；Key Laboratory of Social Computing and Cognitive Intelligence (Dalian University of Technology), Ministry of Education（社会科学与认知智能重点实验室（大连理工大学））；Department of Automation, Tsinghua University（清华大学自动化系）
As an important application form of immersive multimedia services, free-viewpoint video(FVV) enables users with great immersive experience by strong interaction. However, the computational complexity of virtual view synthesis algorithms poses a significant challenge to the real-time performance of an FVV system. Furthermore, the individuality of user interaction makes it difficult to serve multiple users simultaneously for a system with conventional architecture. In this paper, we novelly introduce a CNN-based view interpolation algorithm to synthesis dense virtual views in real time. Based on…
Struct-GStream: Towards Efficient Free-Viewpoint Video Streaming at Low-Bitrates with Structured 3D Gaussians
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Chinese Academy of Sciences（中国科学院）；University of Science and Technology Beijing（北京科技大学）
Ecole Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）；National Institute of Informatics（国立信息学研究所）
Enabling users to interactively navigate through different viewpoints of a static scene is a new interesting functionality in 3D streaming systems. While it opens exciting perspectives towards rich multimedia applications, it requires the design of novel representations and coding techniques in order to solve the new challenges imposed by interactive navigation. Interactivity clearly brings new design constraints: the encoder is unaware of the exact decoding process, while the decoder has to reconstruct information from incomplete subsets of data since the server can generally not transmit ima…
LOCI: Spatial Linear Memory for Streaming World Models
Institute of Foundation Models, Mohamed bin Zayed University of Artificial Intelligence（基础模型研究所，穆罕默德·本·扎耶德人工智能大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Pinscreen
HiCoM: Hierarchical Coherent Motion for Streamable Dynamic Scene with 3D Gaussian Splatting
School of Electronic and Computer Engineering, Peking University（北京大学电子与计算机工程学院）；Pengcheng Laboratory（鹏城实验室）；Peking University Shenzhen Graduate School（北京大学深圳研究生院）
ReCon-GS: Continuum-Preserved Gaussian Streaming for Fast and Compact Reconstruction of Dynamic Scenes
School of Electronic and Computer Engineering, Peking University（电子与计算机工程学院，北京大学）；National Engineering Research Center of Visual Technology, Peking University（视觉技术国家工程研究中心，北京大学）；Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology, Peking University（广东省超高定义沉浸媒体技术重点实验室，北京大学）
