基于自监督学习的真实世界双缩放超分辨率算法
Self-Supervised Learning for Real-World Super-Resolution from Dual Zoomed Observations
摘要:
在本文中,我们考虑了基于参考的超分辨率(RefSR)中的两个具有挑战性的问题,(i)如何选择合适的参考图像,以及(ii)如何以自监督的方式学习真实世界的RefSR。特别是,我们提出了一种新的自监督学习方法,从双摄像机变焦(SelfDoppler R)的观察真实世界的图像SR。考虑到多个相机在现代智能手机中的普及,更变焦(长焦)的图像可以自然地被用作参考以引导更少变焦(短焦)的图像的SR。此外,SelfDoppler R学习深度网络以获得短焦图像的SR结果,以具有与长焦图像相同的分辨率。为此,我们采取的长焦图像,而不是一个额外的高分辨率图像作为监督信息,并从中选择一个中心补丁作为参考,超分辨率相应的短焦图像补丁。为了减轻短焦低分辨率(LR)图像和长焦地面实况(GT)图像之间的不对准的影响,我们设计了一个伪LR生成器,并将GT映射到伪LR,同时保持空间位置不变。然后利用自适应空间Transformer网络(AdaR)对LR特征进行变形,并将Ref特征与GT进行匹配。在测试过程中,SelfDADaR可以直接部署,以长焦图像为参考对整个短焦图像进行超求解。实验表明,我们的方法实现了更好的定量和定性性能对国家的艺术。代码可在https://github.com/cszhilu1998/SelfDZSR上获得。
英文摘要:
In this paper, we consider two challenging issues in reference-based super-resolution (RefSR), (i) how to choose a proper reference image, and (ii) how to learn real-world RefSR in a self-supervised manner. Particularly, we present a novel self-supervised learning approach for real-world image SR from observations at dual camera zooms (SelfDZSR). Considering the popularity of multiple cameras in modern smartphones, the more zoomed (telephoto) image can be naturally leveraged as the reference to guide the SR of the lesser zoomed (short-focus) image. Furthermore, SelfDZSR learns a deep network to obtain the SR result of short-focus image to have the same resolution as the telephoto image. For this purpose, we take the telephoto image instead of an additional high-resolution image as the supervision information and select a center patch from it as the reference to super-resolve the corresponding short-focus image patch. To mitigate the effect of the misalignment between short-focus low-resolution (LR) image and telephoto ground-truth (GT) image, we design an auxiliary-LR generator and map the GT to an auxiliary-LR while keeping the spatial position unchanged. Then the auxiliary-LR can be utilized to deform the LR features by the proposed adaptive spatial transformer networks (AdaSTN), and match the Ref features to GT. During testing, SelfDZSR can be directly deployed to super-solve the whole short-focus image with the reference of telephoto image. Experiments show that our method achieves better quantitative and qualitative performance against state-of-the-arts. Codes are available at https://github.com/cszhilu1998/SelfDZSR.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。