arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.19202cs.CV
收藏

UniE2F: 一种基于视频基础模型的统一扩散框架用于事件到帧重建

UniE2F: A Unified Diffusion Framework for Event-to-Frame Reconstruction with Video Foundation Models

Department of Computer Science, City University of Hong Kong(香港城市大学计算机科学系);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东人工智能与数字经济实验室(深圳));Department of Computer Science, City University of Hong Kong (Dongguan)(香港城市大学(东莞)计算机科学系)

Gang Xu, Zhiyu Zhu, Junhui Hou

AI总结:

本文提出UniE2F框架,利用预训练视频扩散模型的生成先验,从稀疏事件数据中重建高保真视频帧,通过引入事件基帧间残差引导提升重建精度,并扩展到零样本视频帧插值与预测。

摘要:

事件相机在高速、低功耗和高动态范围场景感知方面表现出色。然而,由于其仅记录相对强度变化而非绝对强度,导致的数据流存在显著的空间信息和静态纹理细节丢失。本文通过利用预训练视频扩散模型的生成先验,从稀疏事件数据中重建高保真视频帧。首先,通过直接应用事件数据作为条件合成视频建立基线模型。然后,基于事件流与视频帧之间的物理相关性,引入事件基帧间残差引导以增强视频帧重建的准确性。进一步,通过调节反向扩散采样过程,将方法扩展到零样本视频帧插值和预测,从而创建统一的事件到帧重建框架。实验结果表明,本文方法在真实世界和合成数据集上均优于先前方法。我们还建议评审人员参考补充材料中的视频演示以获取视频结果。代码将在https://github.com/CS-GangXu/UniE2F上公开。

英文摘要:

Event cameras excel at high-speed, low-power, and high-dynamic-range scene perception. However, as they fundamentally record only relative intensity changes rather than absolute intensity, the resulting data streams suffer from a significant loss of spatial information and static texture details. In this paper, we address this limitation by leveraging the generative prior of a pre-trained video diffusion model to reconstruct high-fidelity video frames from sparse event data. Specifically, we first establish a baseline model by directly applying event data as a condition to synthesize videos. Then, based on the physical correlation between the event stream and video frames, we further introduce the event-based inter-frame residual guidance to enhance the accuracy of video frame reconstruction. Furthermore, we extend our method to video frame interpolation and prediction in a zero-shot manner by modulating the reverse diffusion sampling process, thereby creating a unified event-to-frame reconstruction framework. Experimental results on real-world and synthetic datasets demonstrate that our method outperforms previous reconstruction approaches on most quantitative metrics and achieves superior qualitative results, while achieving competitive zero-shot performance compared with dedicated methods explicitly trained for interpolation and prediction tasks. We also refer the reviewers to the video demo contained in the supplementary material for video results. The code will be publicly available at https://github.com/CS-GangXu/UniE2F.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑