arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1804.02684cs.CVcs.GR
收藏

基于学习的视频运动放大

Learning-based Video Motion Magnification

MIT(麻省理工学院);HBKU QCRI(哈马德·本·哈利法大学卡塔尔计算研究所);Google Research(谷歌研究院)

Tae-Hyun Oh, Ronnachai Jaroensri, Changil Kim, Mohamed Elgharib, Frédo Durand, William T. Freeman, Wojciech Matusik

摘要:

视频运动放大技术使我们能够看到以前肉眼看不到的微小运动,例如振动的飞机机翼或受风影响的摇摆建筑物。因为运动很小,所以放大结果容易出现噪声或过度模糊。现有技术依赖于手工设计的过滤器来提取可能不是最佳的表示。在本文中,我们试图使用深度卷积神经网络直接从示例中学习滤波器。为了使训练易于处理,我们精心设计了一个能够很好地捕捉小运动的合成数据集,并使用两帧输入进行训练。我们表明,学习的滤波器在真实视频上实现了高质量的结果,比以前的方法具有更少的振铃伪影和更好的噪声特性。虽然我们的模型没有使用时间滤波器进行训练,但我们发现时间滤波器可以与我们提取的表示一起使用,放大倍数高达中等,从而实现基于频率的运动选择。最后,我们分析了学习的过滤器,并表明它们的行为类似于以前的作品中使用的衍生过滤器。我们的代码,训练模型和数据集将在线提供。

英文摘要:

Video motion magnification techniques allow us to see small motions previously invisible to the naked eyes, such as those of vibrating airplane wings, or swaying buildings under the influence of the wind. Because the motion is small, the magnification results are prone to noise or excessive blurring. The state of the art relies on hand-designed filters to extract representations that may not be optimal. In this paper, we seek to learn the filters directly from examples using deep convolutional neural networks. To make training tractable, we carefully design a synthetic dataset that captures small motion well, and use two-frame input for training. We show that the learned filters achieve high-quality results on real videos, with less ringing artifacts and better noise characteristics than previous methods. While our model is not trained with temporal filters, we found that the temporal filters can be used with our extracted representations up to a moderate magnification, enabling a frequency-based motion selection. Finally, we analyze the learned filters and show that they behave similarly to the derivative filters used in previous works. Our code, trained model, and datasets will be available online.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑