Puzzle Similarity：一种用于3D场景重建中伪影检测的感知引导交叉参考度量
Puzzle Similarity: A Perceptually-guided Cross-Reference Metric for Artifact Detection in 3D Scene Reconstructions
AI总结：
针对3D重建新视图伪影检测缺乏有效度量的问题，提出交叉参考度量Puzzle Similarity，通过训练视图块分布定位伪影，构建人工标注数据集验证其达到SOTA，可提升相关应用效果。
摘要：
现代重建技术能够从稀疏的2D视图中有效建模复杂3D场景。然而，由于缺乏真实图像（ground truth），且无参考图像度量在生成可靠的伪影图方面存在局限性，自动评估新视图质量并识别伪影颇具挑战。此类度量的缺失阻碍了新视图的质量评估，也限制了修复（inpainting）等后处理技术用于提升重建质量的应用。为解决这一问题，近期研究提出了一类新的度量方法——交叉参考度量，仅利用其他视角采集的上下文信息即可预测图像质量（arXiv:2404.14409）。在本研究中，我们提出了一种全新的交叉参考度量Puzzle Similarity，旨在定位新视图中的伪影。我们的方法利用训练视图的图像块统计信息建立场景特定分布，后续用于识别新视图中重建质量较差的区域。鉴于目前缺乏在3D重建场景下评估交叉参考方法的有效指标，我们收集了一个全新的人工标注数据集，包含未见过的重建视图中的伪影和畸变图。通过该数据集我们证明，即使没有对齐的参考图像，我们的方法在新视图伪影定位上仍能达到当前最优水平，结果与人类评估高度相关。我们还可以利用这一新度量提升自动图像修复、引导式采集或稀疏输入3D重建等应用的效果。项目页面见https://nihermann.github.io/puzzlesim/ 。
英文摘要：
Modern reconstruction techniques can effectively model complex 3D scenes from sparse 2D views. However, automatically assessing the quality of novel views and identifying artifacts is challenging due to the lack of ground truth images and the limitations of no-reference image metrics in predicting reliable artifact maps. The absence of such metrics hinders assessment of the quality of novel views and limits the adoption of post-processing techniques, such as inpainting, to enhance reconstruction quality. To tackle this, recent work has established a new category of metrics (cross-reference), predicting image quality solely by leveraging context from alternate viewpoint captures (arXiv:2404.14409). In this work, we propose a new cross-reference metric, Puzzle Similarity, which is designed to localize artifacts in novel views. Our approach utilizes image patch statistics from the training views to establish a scene-specific distribution, later used to identify poorly reconstructed regions in the novel views. Given the lack of good measures to evaluate cross-reference methods in the context of 3D reconstruction, we collected a novel human-labeled dataset of artifact and distortion maps in unseen reconstructed views. Through this dataset, we demonstrate that our method achieves state-of-the-art localization of artifacts in novel views, correlating with human assessment, even without aligned references. We can leverage our new metric to enhance applications like automatic image restoration, guided acquisition, or 3D reconstruction from sparse inputs. Find the project page at https://nihermann.github.io/puzzlesim/ .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
