arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1804.08758cs.CV
收藏

可切换时间传播网络

Switchable Temporal Propagation Network

NVIDIA(英伟达);UC Merced(加州大学默塞德分校);Dalian University of Technology(大连理工大学)

Sifei Liu, Guangyu Zhong, Shalini De Mello, Jinwei Gu, Varun Jampani, Ming-Hsuan Yang, Jan Kautz

摘要:

视频在帧之间包含高度冗余的信息。这种冗余在视频压缩和编码中已经被广泛研究,但是对于更高级的视频处理较少探索。在本文中,我们提出了一个可学习的统一框架,用于传播视频图像的各种视觉属性,包括但不限于颜色,高动态范围(HDR)和分割信息,其中属性仅适用于少数关键帧。我们的方法是基于一个时间传播网络(TPN),该模型的过渡相关的亲和力之间的一对帧在一个纯粹的数据驱动的方式。我们从理论上证明了TPN的两个基本因素:(a)通过将全局变换矩阵正则化为正交,可以在传播过程中很好地保留属性的“风格能量”;(b)这种正则化可以通过所提出的可切换TPN实现,并在帧对上进行双向训练。我们将可切换TPN应用于三个任务:基于几个颜色关键帧对灰度视频进行着色,从低动态范围(LDR)视频和几个HDR帧生成HDR视频,以及从视频中的第一帧传播分割掩码。实验结果表明,我们的方法是显着更准确和更有效的比国家的最先进的方法。

英文摘要:

Videos contain highly redundant information between frames. Such redundancy has been extensively studied in video compression and encoding, but is less explored for more advanced video processing. In this paper, we propose a learnable unified framework for propagating a variety of visual properties of video images, including but not limited to color, high dynamic range (HDR), and segmentation information, where the properties are available for only a few key-frames. Our approach is based on a temporal propagation network (TPN), which models the transition-related affinity between a pair of frames in a purely data-driven manner. We theoretically prove two essential factors for TPN: (a) by regularizing the global transformation matrix as orthogonal, the "style energy" of the property can be well preserved during propagation; (b) such regularization can be achieved by the proposed switchable TPN with bi-directional training on pairs of frames. We apply the switchable TPN to three tasks: colorizing a gray-scale video based on a few color key-frames, generating an HDR video from a low dynamic range (LDR) video and a few HDR frames, and propagating a segmentation mask from the first frame in videos. Experimental results show that our approach is significantly more accurate and efficient than the state-of-the-art methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑