This paper addresses the video rescaling task, which arises from the needs of adapting the video spatial resolution to suit individual viewing devices. We aim to jointly optimize video downscaling and upscaling as a combined task. Most recent studies focus on image-based solutions, which do not consider temporal information. We present two joint optimization approaches based on invertible neural networks with coupling layers. Our Long Short-Term Memory Video Rescaling Network (LSTM-VRN) leverages temporal information in the low-resolution video to form an explicit prediction of the missing high-frequency information for upscaling. Our Multi-input Multi-output Video Rescaling Network (MIMO-VRN) proposes a new strategy for downscaling and upscaling a group of video frames simultaneously. Not only do they outperform the image-based invertible model in terms of quantitative and qualitative results, but also show much improved upscaling quality than the video rescaling methods without joint optimization. To our best knowledge, this work is the first attempt at the joint optimization of video downscaling and upscaling.
Tsinghua University（清华大学）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；Shenzhen University（深圳大学）；Peng Cheng Laboratory（鹏城实验室）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Tencent（腾讯）
TVRN: Invertible Neural Networks for Compression-Aware Temporal Video Rescaling
MOE Key Laboratory of Brain-Inspired Intelligent Perception and Cognition（教育部脑科学与智能感知认知重点实验室）；University of Science and Technology of China（中国科学技术大学）；Information Science and Technology Institution（信息科学与技术研究所）；MCC Lab（MCC实验室）
