arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.08057cs.CVcs.AIcs.LG
收藏

移动设备上的实时神经光场

Real-Time Neural Light Field on Mobile Devices

Snap Inc.(Snap公司);Northeastern University(东北大学)

Junli Cao, Huan Wang, Pavlo Chemerys, Vladislav Shakhrai, Ju Hu, Yun Fu, Denys Makoviichuk, Sergey Tulyakov, Jian Ren

AI总结:

本文提出一种能在移动设备上实时运行的高效神经光场网络架构,通过低延迟和小体积实现高分辨率渲染,质量接近NeRF且优于MobileNeRF。

摘要:

近期在神经渲染场(NeRF)方面的努力,通过利用隐式神经表示来表示3D场景,在新视角合成上展现了令人印象深刻的结果。由于体渲染的过程,NeRF的推理速度极其缓慢,限制了在移动设备等资源受限的硬件上利用NeRF的应用场景。许多工作致力于降低运行NeRF模型的延迟。然而,大多数工作仍需要高端GPU进行加速或额外的存储内存,这些在移动设备上都是不可用的。另一个新兴方向利用神经光场(NeLF)进行加速,因为一条光线只需一次前向传递即可预测像素颜色。尽管如此,为了达到与NeRF相似的渲染质量,NeLF中的网络被设计为计算密集型,这对移动设备并不友好。在这项工作中,我们提出了一种能在移动设备上实时运行的高效网络用于神经渲染。我们遵循NeLF的设置来训练我们的网络。与现有工作不同,我们引入了一种新颖的网络架构,能在移动设备上高效运行,具有低延迟和小体积的特点,即与MobileNeRF相比节省了$15\times \sim 24\times$的存储。我们的模型实现了高分辨率生成,同时在移动设备上对合成和真实世界场景保持实时推理,例如,在iPhone 13上渲染真实3D场景的一张$1008\times756$图像仅需$18.04$ms。此外,我们实现了与NeRF相似的图像质量,并且在真实世界前向数据集上比MobileNeRF具有更好的质量(PSNR为$26.15$对比$25.91$)。

英文摘要:

Recent efforts in Neural Rendering Fields (NeRF) have shown impressive results on novel view synthesis by utilizing implicit neural representation to represent 3D scenes. Due to the process of volumetric rendering, the inference speed for NeRF is extremely slow, limiting the application scenarios of utilizing NeRF on resource-constrained hardware, such as mobile devices. Many works have been conducted to reduce the latency of running NeRF models. However, most of them still require high-end GPU for acceleration or extra storage memory, which is all unavailable on mobile devices. Another emerging direction utilizes the neural light field (NeLF) for speedup, as only one forward pass is performed on a ray to predict the pixel color. Nevertheless, to reach a similar rendering quality as NeRF, the network in NeLF is designed with intensive computation, which is not mobile-friendly. In this work, we propose an efficient network that runs in real-time on mobile devices for neural rendering. We follow the setting of NeLF to train our network. Unlike existing works, we introduce a novel network architecture that runs efficiently on mobile devices with low latency and small size, i.e., saving $15\times \sim 24\times$ storage compared with MobileNeRF. Our model achieves high-resolution generation while maintaining real-time inference for both synthetic and real-world scenes on mobile devices, e.g., $18.04$ms (iPhone 13) for rendering one $1008\times756$ image of real 3D scenes. Additionally, we achieve similar image quality as NeRF and better quality than MobileNeRF (PSNR $26.15$ vs. $25.91$ on the real-world forward-facing dataset).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑