arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2112.08867cs.CV
收藏

GRAM:用于三维感知图像生成的生成辐射流形

GRAM: Generative Radiance Manifolds for 3D-Aware Image Generation

Tsinghua University(清华大学);Microsoft Research Asia(微软亚洲研究院);USTC(中国科学技术大学)

Yu Deng, Jiaolong Yang, Jianfeng Xiang, Xin Tong

摘要:

3D感知图像生成建模旨在生成具有显式可控相机姿态的3D一致图像。最近的工作通过在非结构化的2D图像上训练神经辐射场(NeRF)生成器显示了有希望的结果,但是仍然不能生成具有精细细节的高度真实感的图像。一个关键的原因是体积表示学习的高存储器和计算成本极大地限制了训练期间用于辐射积分的点样本的数量。不足的采样不仅限制了生成器处理细节的表达能力,而且由于不稳定的蒙特卡罗采样引起的噪声而阻碍了有效的GAN训练。我们提出了一种新的方法,调节点采样和辐射场学习的2D流形,体现为一组学习的隐式表面在3D体积。对于每一条观察光线,我们计算光线与曲面的交点,并累积网络生成的辐射。通过训练和渲染这种辐射流形,我们的生成器可以生成具有逼真细节和强大视觉3D一致性的高质量图像。

英文摘要:

3D-aware image generative modeling aims to generate 3D-consistent images with explicitly controllable camera poses. Recent works have shown promising results by training neural radiance field (NeRF) generators on unstructured 2D images, but still can not generate highly-realistic images with fine details. A critical reason is that the high memory and computation cost of volumetric representation learning greatly restricts the number of point samples for radiance integration during training. Deficient sampling not only limits the expressive power of the generator to handle fine details but also impedes effective GAN training due to the noise caused by unstable Monte Carlo sampling. We propose a novel approach that regulates point sampling and radiance field learning on 2D manifolds, embodied as a set of learned implicit surfaces in the 3D volume. For each viewing ray, we calculate ray-surface intersections and accumulate their radiance generated by the network. By training and rendering such radiance manifolds, our generator can produce high quality images with realistic fine details and strong visual 3D consistency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑