arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2209.07105cs.CVcs.AI
收藏

弥合显式与隐式几何变换用于单图像视图合成

Bridging Implicit and Explicit Geometric Transformation for Single-Image View Synthesis

Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院(KAIST));Twelvelabs

Byeongjun Park, Hyojun Go, Changick Kim

AI总结:

针对单图像视图合成中显式与隐式几何的“跷跷板”权衡问题,提出互补损失函数结合非自回归模型,兼顾重投影保留与视图外补全,速度比自回归方法快约100倍。

摘要:

从单张图像创建新视角已借助先进的自回归模型取得了巨大进展,因为未见区域必须从可见场景内容中推断出来。尽管近期方法能生成高质量的新视角,但仅使用一种显式或隐式3D几何进行合成时,会在两个目标之间存在权衡,我们称之为“跷跷板”问题:1) 保留重投影内容,2) 补全逼真的视图外区域。此外,自回归模型需要相当大的计算成本。本文提出一种单图像视图合成框架,在利用高效的非自回归模型的同时缓解跷跷板问题。受显式方法能很好地保留重投影像素、隐式方法能补全逼真的视图外区域这一特性的启发,我们引入一个损失函数来互补两种渲染器。我们的损失函数促进显式特征改善隐式特征的重投影区域,而隐式特征改善显式特征的视图外区域。借助所提出的架构和损失函数,我们能够缓解跷跷板问题,超越基于自回归的最先进方法,并生成图像的速度约快100倍。我们在RealEstate10K和ACID数据集上通过实验验证了所提方法的效率和有效性。

英文摘要:

Creating novel views from a single image has achieved tremendous strides with advanced autoregressive models, as unseen regions have to be inferred from the visible scene contents. Although recent methods generate high-quality novel views, synthesizing with only one explicit or implicit 3D geometry has a trade-off between two objectives that we call the "seesaw" problem: 1) preserving reprojected contents and 2) completing realistic out-of-view regions. Also, autoregressive models require a considerable computational cost. In this paper, we propose a single-image view synthesis framework for mitigating the seesaw problem while utilizing an efficient non-autoregressive model. Motivated by the characteristics that explicit methods well preserve reprojected pixels and implicit methods complete realistic out-of-view regions, we introduce a loss function to complement two renderers. Our loss function promotes that explicit features improve the reprojected area of implicit features and implicit features improve the out-of-view area of explicit features. With the proposed architecture and loss function, we can alleviate the seesaw problem, outperforming autoregressive-based state-of-the-art methods and generating an image $\approx$100 times faster. We validate the efficiency and effectiveness of our method with experiments on RealEstate10K and ACID datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑