arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2203.15946cs.CVcs.AI
收藏

从阴影中学习神经表征

Towards Learning Neural Representations from Shadows

Massachusetts Institute of Technology(麻省理工学院)

Kushagra Tiwary, Tzofi Klinghoffer, Ramesh Raskar

摘要:

我们提出了一种学习神经阴影场的方法,神经阴影场是仅从场景中存在的阴影中学习的神经场景表示。虽然传统的从阴影中提取形状(SfS)算法从阴影中重建几何形状,但它们假设固定的扫描设置,并且无法推广到复杂场景。另一方面,神经渲染算法依赖于RGB图像之间的光度一致性,但在很大程度上忽略了阴影等物理线索,这些线索已被证明可以提供有关场景的有价值的信息。我们观察到,阴影是一个强大的线索,可以约束神经场景表示学习SfS,甚至优于NeRF重建隐藏的几何。我们提出了一种图形启发的微分方法来渲染准确的阴影与体积渲染,预测阴影图,可以比较地面真实的阴影。即使只是二进制阴影图,我们表明,神经渲染可以本地化的对象和估计粗糙的几何形状。我们的方法表明,稀疏的线索,图像可以用来估计几何形状,使用可微体积绘制。此外,我们的框架是高度概括性的,可以与现有的3D重建技术,否则只使用光度一致性。

英文摘要:

We present a method that learns neural shadow fields which are neural scene representations that are only learnt from the shadows present in the scene. While traditional shape-from-shadow (SfS) algorithms reconstruct geometry from shadows, they assume a fixed scanning setup and fail to generalize to complex scenes. Neural rendering algorithms, on the other hand, rely on photometric consistency between RGB images, but largely ignore physical cues such as shadows, which have been shown to provide valuable information about the scene. We observe that shadows are a powerful cue that can constrain neural scene representations to learn SfS, and even outperform NeRF to reconstruct otherwise hidden geometry. We propose a graphics-inspired differentiable approach to render accurate shadows with volumetric rendering, predicting a shadow map that can be compared to the ground truth shadow. Even with just binary shadow maps, we show that neural rendering can localize the object and estimate coarse geometry. Our approach reveals that sparse cues in images can be used to estimate geometry using differentiable volumetric rendering. Moreover, our framework is highly generalizable and can work alongside existing 3D reconstruction techniques that otherwise only use photometric consistency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑