arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2106.07286cs.CV
收藏

TimeLens:基于事件的视频帧插值

TimeLens: Event-based Video Frame Interpolation

Huawei Technologies(华为技术);Zurich Research Center(苏黎世研究中心);Univ. of Zurich(苏黎世大学);ETH Zurich(苏黎世联邦理工学院)

Stepan Tulyakov, Daniel Gehrig, Stamatios Georgoulis, Julius Erbach, Mathias Gehrig, Yuanyou Li, Davide Scaramuzza

摘要:

现有技术的帧插值方法通过从连续的关键帧推断图像中的对象运动来生成中间帧。在没有附加信息的情况下,必须使用一阶近似,即光流,但这种选择限制了可以建模的运动类型,导致高度动态场景中的误差。事件摄像机是一种新型传感器,它通过在帧之间的盲时间提供辅助视觉信息来解决这一限制。它们异步测量每个像素的亮度变化,并以高时间分辨率和低延迟来实现。基于事件的帧插值方法通常采用基于合成的方法,其中预测的帧残差直接应用于关键帧。然而,虽然这些方法可以捕获非线性运动,但它们遭受重影,并且在具有很少事件的低纹理区域中表现不佳。因此,基于综合的方法和基于流程的方法是互补的。在这项工作中,我们介绍了时间透镜,一种新的表示相等的贡献的方法,利用两者的优点。我们广泛地评估我们的方法在三个合成和两个真正的基准,我们表现出高达5.21 dB的改善,在PSNR方面超过最先进的基于帧和基于事件的方法。最后,我们在高度动态的场景中发布了一个新的大规模数据集,旨在推动现有方法的极限。

英文摘要:

State-of-the-art frame interpolation methods generate intermediate frames by inferring object motions in the image from consecutive key-frames. In the absence of additional information, first-order approximations, i.e. optical flow, must be used, but this choice restricts the types of motions that can be modeled, leading to errors in highly dynamic scenarios. Event cameras are novel sensors that address this limitation by providing auxiliary visual information in the blind-time between frames. They asynchronously measure per-pixel brightness changes and do this with high temporal resolution and low latency. Event-based frame interpolation methods typically adopt a synthesis-based approach, where predicted frame residuals are directly applied to the key-frames. However, while these approaches can capture non-linear motions they suffer from ghosting and perform poorly in low-texture regions with few events. Thus, synthesis-based and flow-based approaches are complementary. In this work, we introduce Time Lens, a novel indicates equal contribution method that leverages the advantages of both. We extensively evaluate our method on three synthetic and two real benchmarks where we show an up to 5.21 dB improvement in terms of PSNR over state-of-the-art frame-based and event-based methods. Finally, we release a new large-scale dataset in highly dynamic scenarios, aimed at pushing the limits of existing methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑