Videos contain highly redundant information between frames. Such redundancy has been extensively studied in video compression and encoding, but is less explored for more advanced video processing. In this paper, we propose a learnable unified framework for propagating a variety of visual properties of video images, including but not limited to color, high dynamic range (HDR), and segmentation information, where the properties are available for only a few key-frames. Our approach is based on a temporal propagation network (TPN), which models the transition-related affinity between a pair of frames in a purely data-driven manner. We theoretically prove two essential factors for TPN: (a) by regularizing the global transformation matrix as orthogonal, the "style energy" of the property can be well preserved during propagation; (b) such regularization can be achieved by the proposed switchable TPN with bi-directional training on pairs of frames. We apply the switchable TPN to three tasks: colorizing a gray-scale video based on a few color key-frames, generating an HDR video from a low dynamic range (LDR) video and a few HDR frames, and propagating a segmentation mask from the first frame in videos. Experimental results show that our approach is significantly more accurate and efficient than the state-of-the-art methods.
TiS-TSL: Image-Label Supervised Surgical Video Stereo Matching via Time-Switchable Teacher-Student Learning
Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology(华中科技大学光电实验室);Wuhan United Imaging Surgical Co., Ltd.(武汉联合影像手术有限公司)
Bidirectional Temporal Information Propagation for Moving Infrared Small Target Detection
School of Computer Science and Engineering, University of Electronic Science and Technology of China(电子科技大学计算机科学与工程学院);School of Control Science and Engineering, Shandong University(山东大学控制科学与工程学院)
Weakly Supervised Temporal Action Localization via Representative Snippet Knowledge Propagation
The Chinese University of Hong Kong(香港中文大学);Centre for Perceptual and Interactive Intelligence(感知与智能交互中心);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Error-Propagation-Free Learned Video Compression With Dual-Domain Progressive Temporal Alignment
Department of Electronic Engineering, Shanghai Jiao Tong University(上海交通大学电子工程系);Department of Computer Science and Engineering, Shanghai Jiao Tong University(上海交通大学计算机科学与工程系);College of Information Science and Electronic Engineering, Zhejiang University(浙江大学信息科学与电子工程学院);Naval Aviation University(海军航空大学)
Catalyst4D: High-Fidelity 3D-to-4D Scene Editing via Dynamic Propagation
State Key Laboratory of Complex and Critical Software Environment, Beihang University(北京航空航天大学复杂关键软件环境国家重点实验室);School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院);School of Artificial Intelligence, Beihang University(北京航空航天大学人工智能学院);Zhejiang Industrial Big Data and Robot Intelligent System Key Laboratory, Hangzhou Innovation Institute, Beihang University(北京航空航天大学杭州创新研究院浙江省工业大数据与机器人智能系统重点实验室)
QueryProp: Object Query Propagation for High-Performance Video Object Detection
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);CAS Center for Excellence in Brain Science and Intelligence Technology(中国科学院脑科学与智能技术卓越创新中心)
Temporally Consistent Video Colorization with Deep Feature Propagation and Self-regularization Learning
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);Nanyang Technological University(南洋理工大学);Applied Research Center (ARC), Tencent PCG(腾讯平台与内容事业群应用研究中心)