TimeRewind：利用图像与事件视频扩散倒回时间
TimeRewind: Rewinding Time with Image-and-Events Video Diffusion
AI总结：
本文提出TimeRewind框架，结合事件相机数据与图像到视频扩散模型，通过事件运动适配器从单张静态图像生成连贯的倒回时间视频，解决恢复按下快门前错过瞬间的挑战。
摘要：
本文探讨了从单张拍摄图像“倒回”时间以恢复按下快门键之前错过的短暂瞬间这一全新挑战。由于潜在像素运动的自由度极高，从单一静态帧预测合理的拍摄前运动是一个本质上不适定的任务，这给计算机视觉和计算摄影带来了重大挑战。我们通过利用新兴的神经形态事件相机技术（能以高时间分辨率捕获运动信息），并将该数据与先进的图像到视频扩散模型相结合，克服了这一难题。我们提出的框架引入了一个以事件相机数据为条件的事件运动适配器，引导扩散模型生成在视觉上连贯且在物理上基于所捕获事件的视频。通过大量实验，我们证明了该方法能够合成高质量视频，有效实现了时间“倒回”，展示了将事件相机技术与生成模型相结合的潜力。本研究为计算机视觉、计算摄影与生成建模交叉领域的研究开辟了新途径，为捕捉错过的瞬间并增强未来的消费级相机和智能手机提供了前瞻性解决方案。请访问项目页面 https://timerewind.github.io/ 查看视频结果和代码发布。
英文摘要：
This paper addresses the novel challenge of ``rewinding'' time from a single captured image to recover the fleeting moments missed just before the shutter button is pressed. This problem poses a significant challenge in computer vision and computational photography, as it requires predicting plausible pre-capture motion from a single static frame, an inherently ill-posed task due to the high degree of freedom in potential pixel movements. We overcome this challenge by leveraging the emerging technology of neuromorphic event cameras, which capture motion information with high temporal resolution, and integrating this data with advanced image-to-video diffusion models. Our proposed framework introduces an event motion adaptor conditioned on event camera data, guiding the diffusion model to generate videos that are visually coherent and physically grounded in the captured events. Through extensive experimentation, we demonstrate the capability of our approach to synthesize high-quality videos that effectively ``rewind'' time, showcasing the potential of combining event camera technology with generative models. Our work opens new avenues for research at the intersection of computer vision, computational photography, and generative modeling, offering a forward-thinking solution to capturing missed moments and enhancing future consumer cameras and smartphones. Please see the project page at https://timerewind.github.io/ for video results and code release.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
