Inspired by the recent advances of image super-resolution using convolutional neural network (CNN), we propose a CNN-based block up-sampling scheme for intra frame coding. A block can be down-sampled before being compressed by normal intra coding, and then up-sampled to its original resolution. Different from previous studies on down/up-sampling-based coding, the up-sampling methods in our scheme have been designed by training CNN instead of hand-crafted. We explore a new CNN structure for up-sampling, which features deconvolution of feature maps, multi-scale fusion, and residue learning, making the network both compact and efficient. We also design different networks for the up-sampling of luma and chroma components, respectively, where the chroma up-sampling CNN utilizes the luma information to boost its performance. In addition, we design a two-stage up-sampling process, the first stage being within the block-by-block coding loop, and the second stage being performed on the entire frame, so as to refine block boundaries. We also empirically study how to set the coding parameters of down-sampled blocks for pursuing the frame-level rate-distortion optimization. Our proposed scheme is implemented into the High Efficiency Video Coding (HEVC) reference software, and a comprehensive set of experiments have been performed to evaluate our methods. Experimental results show that our scheme achieves significant bits saving compared with HEVC anchor especially at low bit rates, leading to on average 5.5% BD-rate reduction on common test sequences and on average 9.0% BD-rate reduction on ultra high definition (UHD) test sequences.
Deep Learning-Based Intra Mode Derivation for Versatile Video Coding
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Ningbo University(宁波大学);City University of Hong Kong(香港城市大学)
在帧内编码中,执行率失真优化(RDO)以从预定义的候选列表中实现最佳帧内模式。除了残差信号之外,还需要对最优帧内模式进行编码并发送到解码器侧,在解码器侧消耗大量编码比特。为了进一步提高通用视频编码(Versatile Video Coding,VVC)中帧内编码的性能,提出了一种基于深度学习的帧内模式推导方法(Deep Learning based Intra Mode Derivation,DLIMD)。具体地,帧内模式推导的过程被公式化为多类分类任务,其旨在跳过用于编码比特减少的帧内模式信令的模块。DLIMD的体系结构被开发以适应不同的量化参数设置和可变编码块,包括非正方形的,这是由一个单一的训练模型处理。与现有的基于深度学习的分类问题不同,除了从特征学习网络学习的特征之外,手工制作的特征也被馈送到帧内模式推导网络中。为了与传统方法竞争,在视频编解码器中使用一个额外的二进制标志来指示所选择的RDO方案。大量的实验结果表明,在VVC测试模型平台上,该方法对Y、U和V分量的平均码率分别降低了2.28%、1.74%和2.18%比特率,优于现有的方法。
Dual Learning-based Video Coding with Inception Dense Blocks
Fudan University(复旦大学);Waseda University(早稻田大学)
本文针对PCS Grand Challenge提出了一种基于双重学习的帧内编码方法。该方法主要由帧内预测和重构滤波两部分组成。它们使用不同的网络结构,基于神经网络的帧内预测使用全连接网络来预测块,而基于神经网络的重构滤波使用卷积网络。与以往的滤波工作不同的是,我们在重构滤波网络中使用了具有更强大特征提取能力的网络。并且滤波单元为块级,从而实现更精确的滤波补偿。据我们所知,在所有基于学习的方法中,这是第一次尝试在一个应用中结合两个不同的网络,并且我们在HEVC测试序列上实现了AI配置的最新性能。实验结果表明,与HM-16.20基线相比,我们的方法对于提供的8个序列导致显著的BD速率节省(对于所有帧内和随机访问编码,分别平均降低10.24%和3.57%的比特率)。对于HEVC测试序列,与HM-16.20基线相比,我们的模型还实现了9.70%的BD速率节省。