基于隐式辐射场的分辨率无关神经压缩用于高保真人像视频会议
Resolution-Agnostic Neural Compression for High-Fidelity Portrait Video Conferencing via Implicit Radiance Fields
AI总结：
本文提出基于隐式辐射场的分辨率无关神经压缩方法，通过动态神经辐射场编码表情特征并以体渲染重建人像，实现超低带宽且高保真的视频会议。
摘要：
视频会议近年来受到越来越多的关注。高保真和低带宽是视频会议应用中视频压缩的两大主要目标。大多数开创性方法依赖经典视频压缩编解码器，缺乏高层特征嵌入，因此无法达到极低带宽。近期工作则采用基于模型的神经压缩，利用每帧的稀疏表示（如面部关键点信息）获得超低比特率，但这些方法由于基于二维图像的变形而无法保持高保真。本文提出一种新颖的低带宽神经压缩方法，利用隐式辐射场实现高保真人像视频会议，同时达成两大主要目标。我们利用动态神经辐射场重建具有表情特征的高保真说话人头像，并将其表示为用于传输的帧替代。整个系统在发送端使用深度模型编码表情特征，在接收端以体渲染作为解码器重建人像，从而实现超低带宽。特别是，基于神经辐射场模型的特点，我们的压缩方法与分辨率无关，这意味着我们方法所实现的低带宽与视频分辨率无关，同时在更高分辨率重建中保持保真度。实验结果表明，我们提出的新框架能够（1）构建超低带宽视频会议，（2）保持高保真人像，以及（3）在高分辨率视频压缩方面比以往工作具有更好的性能。
英文摘要：
Video conferencing has caught much more attention recently. High fidelity and low bandwidth are two major objectives of video compression for video conferencing applications. Most pioneering methods rely on classic video compression codec without high-level feature embedding and thus can not reach the extremely low bandwidth. Recent works instead employ model-based neural compression to acquire ultra-low bitrates using sparse representations of each frame such as facial landmark information, while these approaches can not maintain high fidelity due to 2D image-based warping. In this paper, we propose a novel low bandwidth neural compression approach for high-fidelity portrait video conferencing using implicit radiance fields to achieve both major objectives. We leverage dynamic neural radiance fields to reconstruct high-fidelity talking head with expression features, which are represented as frame substitution for transmission. The overall system employs deep model to encode expression features at the sender and reconstruct portrait at the receiver with volume rendering as decoder for ultra-low bandwidth. In particular, with the characteristic of neural radiance fields based model, our compression approach is resolution-agnostic, which means that the low bandwidth achieved by our approach is independent of video resolution, while maintaining fidelity for higher resolution reconstruction. Experimental results demonstrate that our novel framework can (1) construct ultra-low bandwidth video conferencing, (2) maintain high fidelity portrait and (3) have better performance on high-resolution video compression than previous works.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
