arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2104.01772cs.CV
收藏

卷积神经不透明度辐射场

Convolutional Neural Opacity Radiance Fields

ShanghaiTech University(上海科技大学);Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences(中国科学院上海微系统与信息技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Shanghai Engineering Research Center of Intelligent Vision and Imaging(上海智能视觉与成像工程技术研究中心)

Haimin Luo, Anpei Chen, Qixuan Zhang, Bai Pang, Minye Wu, Lan Xu, Jingyi Yu

摘要:

具有复杂不透明度的模糊对象的照片级真实感建模和渲染对于许多沉浸式VR/AR应用至关重要,但它受到强烈的视图依赖性亮度,颜色。在本文中,我们提出了一种新的方案来生成模糊对象的卷积神经渲染器的不透明度辐射场,这是第一个结合显式不透明度监督和卷积机制到神经辐射场框架,使高质量的外观和全局一致的阿尔法蒙版生成在任意新颖的视图。更具体地说,我们提出了一种有效的采样策略,以及相机射线和图像平面,这使得有效的辐射场采样和学习,在一个补丁的方式,以及一种新的体积特征集成方案,生成每补丁混合特征嵌入重建视图一致的精细细节的外观和不透明度输出。我们还采用了一个补丁式的对抗训练计划,以保持高频外观和不透明的细节在一个自我监督的框架。我们还介绍了一个有效的多视图图像捕获系统,以捕捉高品质的颜色和阿尔法地图的挑战模糊的对象。在现有的和我们新的具有挑战性的模糊对象数据集上的大量实验表明,我们的方法实现了照片般逼真,全局一致,精细的详细外观和不透明度自由视点绘制各种模糊对象。

英文摘要:

Photo-realistic modeling and rendering of fuzzy objects with complex opacity are critical for numerous immersive VR/AR applications, but it suffers from strong view-dependent brightness, color. In this paper, we propose a novel scheme to generate opacity radiance fields with a convolutional neural renderer for fuzzy objects, which is the first to combine both explicit opacity supervision and convolutional mechanism into the neural radiance field framework so as to enable high-quality appearance and global consistent alpha mattes generation in arbitrary novel views. More specifically, we propose an efficient sampling strategy along with both the camera rays and image plane, which enables efficient radiance field sampling and learning in a patch-wise manner, as well as a novel volumetric feature integration scheme that generates per-patch hybrid feature embeddings to reconstruct the view-consistent fine-detailed appearance and opacity output. We further adopt a patch-wise adversarial training scheme to preserve both high-frequency appearance and opacity details in a self-supervised framework. We also introduce an effective multi-view image capture system to capture high-quality color and alpha maps for challenging fuzzy objects. Extensive experiments on existing and our new challenging fuzzy object dataset demonstrate that our method achieves photo-realistic, globally consistent, and fined detailed appearance and opacity free-viewpoint rendering for various fuzzy objects.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑