arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.09581cs.CV
收藏

基于C2-Matching的图像和视频参考超分辨率

Reference-based Image and Video Super-Resolution via C2-Matching

Nanyang Technological University(南洋理工大学);Tencent PCG(腾讯平台与内容事业群)

Yuming Jiang, Kelvin C. K. Chan, Xintao Wang, Chen Change Loy, Ziwei Liu

摘要:

基于参考的超分辨率(Ref-SR)最近已经成为通过引入附加的高分辨率(HR)参考图像来增强低分辨率(LR)输入图像或视频的有前景的范例。现有的Ref-SR方法大多依赖于隐式对应匹配从参考图像中借用HR纹理来补偿输入图像中的信息损失。然而,由于输入图像和参考图像之间的两个间隙,执行局部转移是困难的:变换间隙(例如,缩放和旋转)和分辨率间隙(例如,HR和LR)。为了应对这些挑战,我们提出了C2-Matching,它执行显式的鲁棒匹配交叉变换和分辨率。1)为了弥合变换差距,我们提出了一个对比对应网络,它使用输入图像的增强视图来学习变换鲁棒对应。2)为了解决分辨率差距,我们采用师生相关蒸馏,从更容易的HR-HR匹配中提取知识,以指导更模糊的LR-HR匹配。3)最后,我们设计了一个动态的聚合模块,以解决潜在的输入图像和参考图像之间的不对齐问题。此外,为了忠实地评估基于参考的图像超分辨率在现实环境下的性能,我们贡献了Webly参考SR(WR-SR)数据集,模仿实际使用场景。我们还将C2匹配扩展到基于参考的视频超分辨率任务,其中在相似场景中拍摄的图像用作HR参考图像。大量的实验表明,我们提出的C2-Matching显着优于标准CUFED 5基准的艺术状态,也提高了视频SR的性能,通过将C2-Matching组件到视频SR管道。

英文摘要:

Reference-based Super-Resolution (Ref-SR) has recently emerged as a promising paradigm to enhance a low-resolution (LR) input image or video by introducing an additional high-resolution (HR) reference image. Existing Ref-SR methods mostly rely on implicit correspondence matching to borrow HR textures from reference images to compensate for the information loss in input images. However, performing local transfer is difficult because of two gaps between input and reference images: the transformation gap (e.g., scale and rotation) and the resolution gap (e.g., HR and LR). To tackle these challenges, we propose C2-Matching in this work, which performs explicit robust matching crossing transformation and resolution. 1) To bridge the transformation gap, we propose a contrastive correspondence network, which learns transformation-robust correspondences using augmented views of the input image. 2) To address the resolution gap, we adopt teacher-student correlation distillation, which distills knowledge from the easier HR-HR matching to guide the more ambiguous LR-HR matching. 3) Finally, we design a dynamic aggregation module to address the potential misalignment issue between input images and reference images. In addition, to faithfully evaluate the performance of Reference-based Image Super-Resolution under a realistic setting, we contribute the Webly-Referenced SR (WR-SR) dataset, mimicking the practical usage scenario. We also extend C2-Matching to Reference-based Video Super-Resolution task, where an image taken in a similar scene serves as the HR reference image. Extensive experiments demonstrate that our proposed C2-Matching significantly outperforms state of the arts on the standard CUFED5 benchmark and also boosts the performance of video SR by incorporating the C2-Matching component into Video SR pipelines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑