arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.09909cs.CV
收藏

TMP:面向在线视频超分辨率的时序运动传播

TMP: Temporal Motion Propagation for Online Video Super-Resolution

The Hong Kong Polytechnic University(香港理工大学);The Hong Kong University of Science and Technology(香港科技大学)

Zhengqiang Zhang, Ruihuang Li, Shi Guo, Yang Cao, Lei Zhang

AI总结:

本文提出时序运动传播方法TMP,利用相邻帧运动场连续性传播并细化偏移量,结合空间加权对齐,在保证在线视频超分辨率精度的同时显著提升推理速度。

摘要:

在线视频超分辨率(online-VSR)高度依赖有效的对齐模块来聚合时序信息,而严格的延迟要求使得准确且高效的对齐极具挑战性。尽管已取得诸多进展,现有大多数在线VSR方法仍逐帧单独估计运动场以进行对齐,这在计算上是冗余的,并且忽略了相邻帧运动场具有相关性这一事实。在本工作中,我们提出了一种高效的时序运动传播(Temporal Motion Propagation,TMP)方法,利用运动场的连续性在连续帧之间实现快速的像素级对齐。具体而言,我们首先将偏移量从先前帧传播到当前帧,然后在邻域内对其进行细化,这显著缩小了匹配空间并加快了偏移估计过程。此外,为增强对齐的鲁棒性,我们对变形后的特征进行空间加权,其中偏移更精确的位置被赋予更高的重要性。在基准数据集上的实验表明,所提出的TMP方法在在线VSR精度和推理速度上均达到领先水平。TMP的源代码可在 https://github.com/xtudbxk/TMP 获取。

英文摘要:

Online video super-resolution (online-VSR) highly relies on an effective alignment module to aggregate temporal information, while the strict latency requirement makes accurate and efficient alignment very challenging. Though much progress has been achieved, most of the existing online-VSR methods estimate the motion fields of each frame separately to perform alignment, which is computationally redundant and ignores the fact that the motion fields of adjacent frames are correlated. In this work, we propose an efficient Temporal Motion Propagation (TMP) method, which leverages the continuity of motion field to achieve fast pixel-level alignment among consecutive frames. Specifically, we first propagate the offsets from previous frames to the current frame, and then refine them in the neighborhood, which significantly reduces the matching space and speeds up the offset estimation process. Furthermore, to enhance the robustness of alignment, we perform spatial-wise weighting on the warped features, where the positions with more precise offsets are assigned higher importance. Experiments on benchmark datasets demonstrate that the proposed TMP method achieves leading online-VSR accuracy as well as inference speed. The source code of TMP can be found at https://github.com/xtudbxk/TMP.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑