arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1902.09680cs.CV
收藏

事件驱动的视频帧合成

Event-driven Video Frame Synthesis

Northwestern University(西北大学);Peking University(北京大学)

Zihao W. Wang, Weixin Jiang, Kuan He, Boxin Shi, Aggelos Katsaggelos, Oliver Cossairt

摘要:

时域视频帧合成(TVFS)旨在合成不同于已有帧的新帧,在视频编解码、编辑和分析中有着广泛的应用。在本文中,我们提出了一个高帧率TVFS框架,它采取混合输入数据从低速帧为基础的传感器和高速事件为基础的传感器。与基于帧的传感器相比,基于事件的传感器以非常高的速度报告亮度变化,这可以很好地为高帧率TVFS提供有用的时空信息。在我们的框架中,我们首先引入了一个可微的前向模型来近似物理感知过程,融合了两种不同的数据模式,并统一了各种TVFS任务,即,内插、预测和运动去模糊。我们利用自动微分,将测量数据上定义的损失梯度传播回潜在的高帧率视频。我们展示了与最先进技术相比具有更好性能的结果。其次,我们开发了一种基于深度学习的策略来增强第一步的结果,我们将其称为残差“去噪”过程。我们训练的“降噪器”超越了高斯降噪,并显示出对比度增强和运动感知等特性。我们表明,我们的框架是能够处理具有挑战性的场景,包括快速运动和强遮挡。

英文摘要:

Temporal Video Frame Synthesis (TVFS) aims at synthesizing novel frames at timestamps different from existing frames, which has wide applications in video codec, editing and analysis. In this paper, we propose a high framerate TVFS framework which takes hybrid input data from a low-speed frame-based sensor and a high-speed event-based sensor. Compared to frame-based sensors, event-based sensors report brightness changes at very high speed, which may well provide useful spatio-temoral information for high framerate TVFS. In our framework, we first introduce a differentiable forward model to approximate the physical sensing process, fusing the two different modes of data as well as unifying a variety of TVFS tasks, i.e., interpolation, prediction and motion deblur. We leverage autodifferentiation which propagates the gradients of a loss defined on the measured data back to the latent high framerate video. We show results with better performance compared to state-of-the-art. Second, we develop a deep learning-based strategy to enhance the results from the first step, which we refer as a residual "denoising" process. Our trained "denoiser" is beyond Gaussian denoising and shows properties such as contrast enhancement and motion awareness. We show that our framework is capable of handling challenging scenes including both fast motion and strong occlusions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑