NeRF-VPT：通过视角提示调优利用神经辐射场学习新视角表示
NeRF-VPT: Learning Novel View Representations with Neural Radiance Fields via View Prompt Tuning
AI总结：
本文提出即插即用的 NeRF-VPT，通过级联视角提示调优将前序渲染 RGB 作为视觉先验，提升新视角合成质量、PSNR 及稀疏输入适应性。
摘要：
神经辐射场（Neural Radiance Fields，NeRF）在新视角合成中取得了显著成功。尽管如此，为新视角生成高质量图像仍是一项关键挑战。现有工作虽已展现出值得肯定的进展，但在捕捉复杂细节、增强纹理以及获得更优的峰值信噪比（Peak Signal-to-Noise Ratio，PSNR）指标方面，仍需要进一步的重点关注和推进。在本工作中，我们提出 NeRF-VPT，这是一种用于新视角合成的创新方法，旨在应对这些挑战。我们提出的 NeRF-VPT 采用级联视角提示调优范式，其中从前序渲染结果中获得的 RGB 信息会作为后续渲染阶段的指导性视觉提示，期望嵌入在提示中的先验知识能够促进渲染图像质量的逐步提升。NeRF-VPT 在每个训练阶段只需从先前阶段的渲染结果中采样 RGB 数据作为先验，而无需依赖额外引导或复杂技术。因此，我们的 NeRF-VPT 具有即插即用特性，可方便地集成到现有方法中。通过在具有挑战性的真实场景基准（如 Realistic Synthetic 360、Real Forward-Facing、Replica dataset 以及一个用户采集数据集）上将 NeRF-VPT 与若干基于 NeRF 的方法进行比较分析，我们证实 NeRF-VPT 显著提升了基线性能，并能够比所有参与比较的最先进方法更熟练地生成更多高质量新视角图像。此外，NeRF-VPT 的级联学习带来了对稀疏输入场景的适应性，从而显著提高了稀疏视角新视角合成的精度。源代码和数据集可在 https://github.com/Freedomcls/NeRF-VPT 获取。
英文摘要：
Neural Radiance Fields (NeRF) have garnered remarkable success in novel view synthesis. Nonetheless, the task of generating high-quality images for novel views persists as a critical challenge. While the existing efforts have exhibited commendable progress, capturing intricate details, enhancing textures, and achieving superior Peak Signal-to-Noise Ratio (PSNR) metrics warrant further focused attention and advancement. In this work, we propose NeRF-VPT, an innovative method for novel view synthesis to address these challenges. Our proposed NeRF-VPT employs a cascading view prompt tuning paradigm, wherein RGB information gained from preceding rendering outcomes serves as instructive visual prompts for subsequent rendering stages, with the aspiration that the prior knowledge embedded in the prompts can facilitate the gradual enhancement of rendered image quality. NeRF-VPT only requires sampling RGB data from previous stage renderings as priors at each training stage, without relying on extra guidance or complex techniques. Thus, our NeRF-VPT is plug-and-play and can be readily integrated into existing methods. By conducting comparative analyses of our NeRF-VPT against several NeRF-based approaches on demanding real-scene benchmarks, such as Realistic Synthetic 360, Real Forward-Facing, Replica dataset, and a user-captured dataset, we substantiate that our NeRF-VPT significantly elevates baseline performance and proficiently generates more high-quality novel view images than all the compared state-of-the-art methods. Furthermore, the cascading learning of NeRF-VPT introduces adaptability to scenarios with sparse inputs, resulting in a significant enhancement of accuracy for sparse-view novel view synthesis. The source code and dataset are available at \url{https://github.com/Freedomcls/NeRF-VPT}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
