arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.04421cs.CV
收藏

用于卫星视频超分辨率的局部-全局时间差分学习

Local-Global Temporal Difference Learning for Satellite Video Super-Resolution

School of Geodesy and Geomatics, Wuhan University(武汉大学测绘学院);Faculty of Computing, Harbin Institute of Technology(哈尔滨工业大学计算机学院);State Key Laboratory of Information Engineering in Surveying, Mapping, and Remote Sensing, Wuhan University(武汉大学测绘遥感信息工程国家重点实验室);Department of Electrical Engineering and the Institute of Communications Engineering, National Tsing Hua University(国立清华大学电子工程系和通信工程研究所)

Yi Xiao, Qiangqiang Yuan, Kui Jiang, Xianyu Jin, Jiang He, Liangpei Zhang, Chia-Wen Lin

AI总结:

本文提出一种局部-全局时间差分学习方法用于卫星视频超分辨率,通过设计短期和长期时间差分模块及差异补偿单元,有效利用帧间时间信息进行补偿,在五个主流卫星数据集上表现优于现有方法。

摘要:

基于光流和基于核的方法在卫星视频超分辨率(VSR)的时间补偿中已被广泛探索。然而,这些技术在规模化或复杂场景中泛化能力较差,尤其是在卫星视频中。本文提出利用明确定义的时间差分进行高效且有效的时间补偿。为充分利用帧内的局部和全局时间信息,我们系统地建模了短期和长期时间差异,因为我们观察到这些差异具有独特且互补的特性。具体而言,我们设计了短期时间差分模块(S-TDM),从相邻帧的RGB差异图中提取局部运动表示,为准确的纹理表示提供更多线索。为探索整个帧序列中的全局依赖性,提出了长期时间差分模块(L-TDM),其中结合并激活了前向和后向片段之间的差异,以指导时间特征的调制,实现全面的全局补偿。此外,我们进一步提出差异补偿单元(DCU),以丰富目标帧的空间分布与时间补偿结果之间的交互,有助于在细化特征以避免错位的同时保持空间一致性。在五个主流视频卫星上进行的严格客观和主观评估表明,我们的方法优于现有先进方法。代码将在https://github.com/XY-boy/LGTD提供

英文摘要:

Optical-flow-based and kernel-based approaches have been extensively explored for temporal compensation in satellite Video Super-Resolution (VSR). However, these techniques are less generalized in large-scale or complex scenarios, especially in satellite videos. In this paper, we propose to exploit the well-defined temporal difference for efficient and effective temporal compensation. To fully utilize the local and global temporal information within frames, we systematically modeled the short-term and long-term temporal discrepancies since we observed that these discrepancies offer distinct and mutually complementary properties. Specifically, we devise a Short-term Temporal Difference Module (S-TDM) to extract local motion representations from RGB difference maps between adjacent frames, which yields more clues for accurate texture representation. To explore the global dependency in the entire frame sequence, a Long-term Temporal Difference Module (L-TDM) is proposed, where the differences between forward and backward segments are incorporated and activated to guide the modulation of the temporal feature, leading to a holistic global compensation. Moreover, we further propose a Difference Compensation Unit (DCU) to enrich the interaction between the spatial distribution of the target frame and temporal compensated results, which helps maintain spatial consistency while refining the features to avoid misalignment. Rigorous objective and subjective evaluations conducted across five mainstream video satellites demonstrate that our method performs favorably against state-of-the-art approaches. Code will be available at https://github.com/XY-boy/LGTD

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑