RealLiFe:基于分层稀疏梯度下降的实时光场重建
RealLiFe: Real-Time Light Field Reconstruction via Hierarchical Sparse Gradient Descent
AI总结:
针对稀疏视图输入的实时光场重建需求,提出基于分层稀疏梯度下降的RealLiFe方法,借助MPI稀疏流形特性实现高质量实时重建,速度远超离线方法且性能优于现有在线方法。
摘要:
随着扩展现实(XR)技术的兴起,从稀疏视图输入进行实时光场重建的需求日益增长。现有方法可分为两类:离线技术虽能生成高质量新视图,但代价是推理/训练时间长;在线方法要么泛化能力不足,要么生成结果不尽如人意。而我们观察到,多平面图像(MPI)的固有稀疏流形能够在保持渲染质量的同时,大幅加速光场重建。基于这一洞见,我们提出了一种新型光场优化方法RealLiFe,它利用所提出的分层稀疏梯度下降(HSGD),能从稀疏输入图像实时生成高质量光场。技术层面,该方法首先通过3D CNN生成场景的粗略MPI,随后仅利用与场景内容对齐的稀疏MPI梯度,在数次迭代中对其进行进一步优化。大量实验表明,我们的方法在视觉质量上与最先进的离线方法相当,速度平均快100倍;与其他在线方法相比,性能更优(PSNR高出约2 dB)。
英文摘要:
With the rise of Extended Reality (XR) technology, there is a growing need for real-time light field reconstruction from sparse view inputs. Existing methods can be classified into offline techniques, which can generate high-quality novel views but at the cost of long inference/training time, and online methods, which either lack generalizability or produce unsatisfactory results. However, we have observed that the intrinsic sparse manifold of Multi-plane Images (MPI) enables a significant acceleration of light field reconstruction while maintaining rendering quality. Based on this insight, we introduce RealLiFe, a novel light field optimization method, which leverages the proposed Hierarchical Sparse Gradient Descent (HSGD) to produce high-quality light fields from sparse input images in real time. Technically, the coarse MPI of a scene is first generated using a 3D CNN, and it is further optimized leveraging only the scene content aligned sparse MPI gradients in a few iterations. Extensive experiments demonstrate that our method achieves comparable visual quality while being 100x faster on average than state-of-the-art offline methods and delivers better performance (about 2 dB higher in PSNR) compared to other online approaches.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。