arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2107.05307cs.CVcs.AIeess.IV
收藏

基于深度学习的4K视频实时超分辨率系统

Real-Time Super-Resolution System of 4K-Video Based on Deep Learning

Southeast University(东南大学);University of Cambridge(剑桥大学)

Yanpeng Cao, Chengcheng Wang, Changjun Song, Yongming Tang, He Li

摘要:

视频超分辨率(VSR)技术在重建低质量视频方面表现出色,避免了基于插值的算法带来的令人不快的模糊效果。然而,巨大的计算复杂度和内存占用阻碍了可扩展性的边缘和运行时推理在现实生活中的应用,特别是对于大规模的VSR任务。本文探讨了实时的【形】VSR系统的可能性,并设计了一种高效的通用VSR网络EGVSR。所提出的EGVSR是基于时空对抗学习的时间一致性。为了追求高达4K分辨率的更快VSR处理能力,本文尝试选择轻量级的网络结构和高效的上采样方法,在保证高视觉质量的情况下减少EGVSR网络所需的计算量。此外,在实际硬件平台上实现了批量归一化计算融合、卷积加速算法等神经网络加速技术,优化了EGVSR网络的推理过程。最后,我们的EGVSR实现了4K@29.61FPS的实时处理能力。与目前最先进的VSR网络TecoGAN相比,我们实现了85.04%的计算密度降低和7.92倍的性能加速。在视觉质量方面,所提出的EGVSR在大多数指标(例如LPIPS、tOF、tLP等)的列表中名列前茅。在公共测试数据集Vid4上,并在整体性能得分上超过其他最先进的方法。这个项目的源代码可以在www.example.com上找到。

英文摘要:

Video super-resolution (VSR) technology excels in reconstructing low-quality video, avoiding unpleasant blur effect caused by interpolation-based algorithms. However, vast computation complexity and memory occupation hampers the edge of deplorability and the runtime inference in real-life applications, especially for large-scale VSR task. This paper explores the possibility of real-time VSR system and designs an efficient and generic VSR network, termed EGVSR. The proposed EGVSR is based on spatio-temporal adversarial learning for temporal coherence. In order to pursue faster VSR processing ability up to 4K resolution, this paper tries to choose lightweight network structure and efficient upsampling method to reduce the computation required by EGVSR network under the guarantee of high visual quality. Besides, we implement the batch normalization computation fusion, convolutional acceleration algorithm and other neural network acceleration techniques on the actual hardware platform to optimize the inference process of EGVSR network. Finally, our EGVSR achieves the real-time processing capacity of 4K@29.61FPS. Compared with TecoGAN, the most advanced VSR network at present, we achieve 85.04% reduction of computation density and 7.92x performance speedups. In terms of visual quality, the proposed EGVSR tops the list of most metrics (such as LPIPS, tOF, tLP, etc.) on the public test dataset Vid4 and surpasses other state-of-the-art methods in overall performance score. The source code of this project can be found on https://github.com/Thmen/EGVSR.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑