基于卷积核旁路移植的在线视频超分辨率
Online Video Super-Resolution with Convolutional Kernel Bypass Graft
摘要：
近年来，基于深度学习的模型在视频超分辨率（VSR）方面取得了显著的性能，但这些模型大多不太适用于在线视频应用。这些方法仅考虑失真质量，而忽略了在线应用的关键要求，例如，低延迟和低模型复杂度。本文主要研究在线视频传输，其中要求采用VSR算法实时的逐帧生成高分辨率的视频序列。为了解决这些挑战，我们提出了一个极低的延迟VSR算法的基础上一种新的内核知识转移方法，命名为卷积核旁路移植（CKBG）。首先，我们设计了一个轻量级的网络结构，不需要未来的帧作为输入，并节省额外的时间成本缓存这些帧。然后，我们提出的CKBG方法通过使用“内核移植”绕过原始网络来增强这个轻量级的基础模型，这些内核移植是包含外部预训练图像SR模型的先验知识的额外卷积内核。在测试阶段，我们通过将嫁接的多分支网络转换为简单的单路径结构来进一步加速嫁接的多分支网络。实验结果表明，我们提出的方法可以处理高达110 FPS的在线视频序列，具有非常低的模型复杂度和竞争力的SR性能。
英文摘要：
Deep learning-based models have achieved remarkable performance in video super-resolution (VSR) in recent years, but most of these models are less applicable to online video applications. These methods solely consider the distortion quality and ignore crucial requirements for online applications, e.g., low latency and low model complexity. In this paper, we focus on online video transmission, in which VSR algorithms are required to generate high-resolution video sequences frame by frame in real time. To address such challenges, we propose an extremely low-latency VSR algorithm based on a novel kernel knowledge transfer method, named convolutional kernel bypass graft (CKBG). First, we design a lightweight network structure that does not require future frames as inputs and saves extra time costs for caching these frames. Then, our proposed CKBG method enhances this lightweight base model by bypassing the original network with ``kernel grafts'', which are extra convolutional kernels containing the prior knowledge of external pretrained image SR models. In the testing phase, we further accelerate the grafted multi-branch network by converting it into a simple single-path structure. Experiment results show that our proposed method can process online video sequences up to 110 FPS, with very low model complexity and competitive SR performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
