arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2204.08563cs.CV
收藏

Cylin-Painting:无缝 360° 全景图像外绘及拓展

Cylin-Painting: Seamless {360\textdegree} Panoramic Image Outpainting and Beyond

Beijing Jiaotong University(北京交通大学);Xi’an Jiaotong University(西安交通大学);Sun Yat-sen University(中山大学)

Kang Liao, Xiangyu Xu, Chunyu Lin, Wenqi Ren, Yunchao Wei, Yao Zhao

AI总结:

本文提出 Cylin-Painting 框架,通过融合图像修复与外绘并引入可学习位置嵌入的圆柱卷积,实现高质量无缝 360° 全景外绘,并可推广到多种全景视觉任务。

摘要:

图像外绘因能够从局部视图生成完整场景而受到越来越多的关注,并为构建 360° 全景图像提供了一种有价值的解决方案。由于图像外绘存在单向补全流这一固有问题,以往方法将原始问题转化为允许双向流的图像修复。然而,我们发现图像修复也有其自身局限,并且在某些情况下不如外绘。如何将二者结合以兼得两者优势,至今仍缺乏充分探索。本文深入分析了图像修复与外绘之间的差异,这种差异本质上取决于在不同空间排列下,源像素如何对未知区域作出贡献。受该分析启发,我们提出了 Cylin-Painting 框架,该框架在图像修复与外绘之间建立有意义的协作,并高效融合不同排列,以期在无缝圆柱体上利用二者的互补优势。尽管如此,直接应用圆柱式卷积通常会产生视觉上不理想的结果,因为它丢弃了重要的位置信息。为解决这一问题,我们进一步提出一种可学习的位置嵌入策略,将缺失的位置编码成分纳入圆柱卷积,从而显著改善全景结果。值得注意的是,尽管所提算法是为图像外绘开发的,但它可以有效扩展到其他全景视觉任务,例如目标检测、深度估计和图像超分辨率。代码将在 https://github.com/KangLiao929/Cylin-Painting 提供。

英文摘要:

Image outpainting gains increasing attention since it can generate the complete scene from a partial view, providing a valuable solution to construct {360\textdegree} panoramic images. As image outpainting suffers from the intrinsic issue of unidirectional completion flow, previous methods convert the original problem into inpainting, which allows a bidirectional flow. However, we find that inpainting has its own limitations and is inferior to outpainting in certain situations. The question of how they may be combined for the best of both has as yet remained under-explored. In this paper, we provide a deep analysis of the differences between inpainting and outpainting, which essentially depends on how the source pixels contribute to the unknown regions under different spatial arrangements. Motivated by this analysis, we present a Cylin-Painting framework that involves meaningful collaborations between inpainting and outpainting and efficiently fuses the different arrangements, with a view to leveraging their complementary benefits on a seamless cylinder. Nevertheless, straightforwardly applying the cylinder-style convolution often generates visually unpleasing results as it discards important positional information. To address this issue, we further present a learnable positional embedding strategy to incorporate the missing component of positional encoding into the cylinder convolution, which significantly improves the panoramic results. It is noted that while developed for image outpainting, the proposed algorithm can be effectively extended to other panoramic vision tasks, such as object detection, depth estimation, and image super-resolution. Code will be made available at \url{https://github.com/KangLiao929/Cylin-Painting}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑