arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2308.07032cs.CVcs.LG
收藏

S3IM:随机结构相似性及其在神经场中不合理的有效性

S3IM: Stochastic Structural SIMilarity and Its Unreasonable Effectiveness for Neural Fields

Baidu Research(百度研究院)

Zeke Xie, Xindi Yang, Yujie Yang, Qi Sun, Yixiang Jiang, Haoran Wang, Yunfeng Cai, Mingming Sun

AI总结:

本文提出S3IM随机结构相似性损失,以非局部多路方式利用像素集合结构监督,几乎无额外成本地显著提升NeRF、神经表面表示及相关方法。

摘要:

近年来,Neural Radiance Field(NeRF,神经辐射场)通过仅使用带位姿的RGB图像学习隐式表示,在渲染给定场景的新视角图像方面取得了巨大成功。NeRF及相关神经场方法(例如神经表面表示)通常优化逐点损失并进行逐点预测,其中一个数据点对应一个像素。遗憾的是,尽管已知图像或场景中的像素能够提供丰富的结构信息,但这一系列研究未能利用远距离像素的集体监督。据我们所知,我们首次通过一种新颖的Stochastic Structural SIMilarity(S3IM,随机结构相似性)损失,为NeRF及相关神经场方法设计了非局部多路训练范式;该损失将多个数据点作为一个整体集合来处理,而不是独立处理多个输入。大量实验表明,S3IM在几乎不增加额外成本的情况下,能够以不合理的有效性改进NeRF和神经表面表示。对于那些相对困难的任务,质量指标的提升可能尤为显著:例如,在八个新视角合成任务中,TensoRF和DVGO的测试MSE损失意外下降了90%以上;在八个表面重建任务中,NeuS的F-score提升198%,Chamfer L1距离降低64%。此外,即使在稀疏输入、受损图像和动态场景下,S3IM也始终保持鲁棒。

英文摘要:

Recently, Neural Radiance Field (NeRF) has shown great success in rendering novel-view images of a given scene by learning an implicit representation with only posed RGB images. NeRF and relevant neural field methods (e.g., neural surface representation) typically optimize a point-wise loss and make point-wise predictions, where one data point corresponds to one pixel. Unfortunately, this line of research failed to use the collective supervision of distant pixels, although it is known that pixels in an image or scene can provide rich structural information. To the best of our knowledge, we are the first to design a nonlocal multiplex training paradigm for NeRF and relevant neural field methods via a novel Stochastic Structural SIMilarity (S3IM) loss that processes multiple data points as a whole set instead of process multiple inputs independently. Our extensive experiments demonstrate the unreasonable effectiveness of S3IM in improving NeRF and neural surface representation for nearly free. The improvements of quality metrics can be particularly significant for those relatively difficult tasks: e.g., the test MSE loss unexpectedly drops by more than 90% for TensoRF and DVGO over eight novel view synthesis tasks; a 198% F-score gain and a 64% Chamfer $L_{1}$ distance reduction for NeuS over eight surface reconstruction tasks. Moreover, S3IM is consistently robust even with sparse inputs, corrupted images, and dynamic scenes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑