arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2102.05822cs.CV
收藏

基于帧差的视频风格化时域丢失算法

Frame Difference-Based Temporal Loss for Video Stylization

Tsinghua University(清华大学);Columbia University(哥伦比亚大学);Rice University(莱斯大学)

Jianjin Xu, Zheyang Xiong, Xiaolin Hu

摘要:

神经风格转换模型已被用于将普通视频风格转换为特定风格。为了确保风格化视频的帧之间的时间不一致性,常见的方法是估计原始视频中的像素的光流,并且使生成的像素与估计的光流匹配。这通过在模型训练期间最小化基于光流(OFB)的损失来实现。然而,光流估计本身是一项具有挑战性的任务,特别是在复杂场景中。此外,它还带来了很高的计算成本。我们提出了一个更简单的时间损失称为帧差为基础(FDB)的损失,以解决时间不一致的问题。它被定义为风格化帧之间的差异与原始帧之间的差异之间的距离。在像素空间和由卷积神经网络指定的特征空间中测量两个帧之间的差异。一组涉及62名受试者的25,600张选票的人类行为实验表明,所提出的FDB损失的性能与OFB损失的性能相匹配。在两种典型的视频风格化模型上,通过对生成视频的稳定性和风格化质量的主观评价来衡量性能。结果表明,提出的FDB损失是一个强大的替代常用的OFB损失的视频风格化。

英文摘要:

Neural style transfer models have been used to stylize an ordinary video to specific styles. To ensure temporal inconsistency between the frames of the stylized video, a common approach is to estimate the optic flow of the pixels in the original video and make the generated pixels match the estimated optical flow. This is achieved by minimizing an optical flow-based (OFB) loss during model training. However, optical flow estimation is itself a challenging task, particularly in complex scenes. In addition, it incurs a high computational cost. We propose a much simpler temporal loss called the frame difference-based (FDB) loss to solve the temporal inconsistency problem. It is defined as the distance between the difference between the stylized frames and the difference between the original frames. The differences between the two frames are measured in both the pixel space and the feature space specified by the convolutional neural networks. A set of human behavior experiments involving 62 subjects with 25,600 votes showed that the performance of the proposed FDB loss matched that of the OFB loss. The performance was measured by subjective evaluation of stability and stylization quality of the generated videos on two typical video stylization models. The results suggest that the proposed FDB loss is a strong alternative to the commonly used OFB loss for video stylization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑