arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2211.01639cs.CV
收藏

视频超分辨率的帧间时间一致性学习

Temporal Consistency Learning of inter-frames for Video Super-Resolution

Beijing Jiaotong University(北京交通大学);University of Science and Technology Beijing(北京科技大学)

Meiqin Liu, Shuo Jin, Chao Yao, Chunyu Lin, Yao Zhao

摘要:

视频超分辨率(VSR)是一项旨在从低分辨率(LR)参考帧和多个相邻帧重建高分辨率(HR)帧的任务。关键的操作是利用当前帧的相对失调帧进行重建,并保持结果的一致性。现有的方法一般探索信息传播和帧对齐,以提高VSR的性能。然而,很少有研究关注帧间的时间一致性。在本文中,我们提出了一个时间一致性学习网络(TCNet)的VSR在端到端的方式,以提高重建视频的一致性。设计了一个时空稳定性模块,用于从帧间学习自对准。特别是,采用相关匹配来利用每个帧的空间依赖性来保持结构的稳定性。此外,自注意力机制被用来学习的时间对应性,以实现自适应扭曲操作的时间一致性之间的多帧。此外,还设计了一个混合经常性架构,以利用短期和长期信息。我们进一步提出了一个渐进式融合模块进行多级融合的时空特征。最后利用融合后的特征对重构帧进行细化。各种实验的客观和主观结果表明,TCNet在不同的基准数据集上具有优越的性能,与几种最先进的方法相比。

英文摘要:

Video super-resolution (VSR) is a task that aims to reconstruct high-resolution (HR) frames from the low-resolution (LR) reference frame and multiple neighboring frames. The vital operation is to utilize the relative misaligned frames for the current frame reconstruction and preserve the consistency of the results. Existing methods generally explore information propagation and frame alignment to improve the performance of VSR. However, few studies focus on the temporal consistency of inter-frames. In this paper, we propose a Temporal Consistency learning Network (TCNet) for VSR in an end-to-end manner, to enhance the consistency of the reconstructed videos. A spatio-temporal stability module is designed to learn the self-alignment from inter-frames. Especially, the correlative matching is employed to exploit the spatial dependency from each frame to maintain structural stability. Moreover, a self-attention mechanism is utilized to learn the temporal correspondence to implement an adaptive warping operation for temporal consistency among multi-frames. Besides, a hybrid recurrent architecture is designed to leverage short-term and long-term information. We further present a progressive fusion module to perform a multistage fusion of spatio-temporal features. And the final reconstructed frames are refined by these fused features. Objective and subjective results of various experiments demonstrate that TCNet has superior performance on different benchmark datasets, compared to several state-of-the-art methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑