IE2Video: 适配预训练扩散模型以实现事件视频重建
IE2Video: Adapting Pretrained Diffusion Models for Event-Based Video Reconstruction
AI总结：
IE2Video通过适配预训练扩散模型，实现从单帧和事件数据重建RGB视频，提升视频重建质量与泛化能力。
摘要：
在监视、机器人和可穿戴系统中进行连续视频监控面临一个基本的功率限制：传统RGB相机通过固定速率捕获消耗大量能量。事件相机提供稀疏、运动驱动的传感，具有低功耗消耗，但产生异步事件流而非RGB视频。我们提出一种混合捕获范式，记录稀疏RGB关键帧并同时连续捕获事件流，然后在离线中重建完整的RGB视频——减少捕获功率消耗的同时保持标准视频输出以供下游应用使用。我们引入了Image and Event to Video（IE2Video）任务：从单一初始帧和后续事件相机数据中重建RGB视频序列。我们研究了两种架构策略：适配自回归模型（HyperE2VID）用于RGB生成，以及通过学习编码器和低秩适应将事件表示注入到预训练的文本到视频扩散模型（LTX）中。我们的实验表明，扩散方法在感知质量上比自回归基线更好（0.283 vs 0.422 LPIPS）。我们验证了我们的方法在三个事件相机数据集（BS-ERGB、HS-ERGB远/近）上在不同序列长度（32-128帧）下的鲁棒性，显示出强大的跨数据集泛化能力，在未见过的捕获配置中表现优异。
英文摘要：
Continuous video monitoring in surveillance, robotics, and wearable systems faces a fundamental power constraint: conventional RGB cameras consume substantial energy through fixed-rate capture. Event cameras offer sparse, motion-driven sensing with low power consumption, but produce asynchronous event streams rather than RGB video. We propose a hybrid capture paradigm that records sparse RGB keyframes alongside continuous event streams, then reconstructs full RGB video offline -- reducing capture power consumption while maintaining standard video output for downstream applications. We introduce the Image and Event to Video (IE2Video) task: reconstructing RGB video sequences from a single initial frame and subsequent event camera data. We investigate two architectural strategies: adapting an autoregressive model (HyperE2VID) for RGB generation, and injecting event representations into a pretrained text-to-video diffusion model (LTX) via learned encoders and low-rank adaptation. Our experiments demonstrate that the diffusion-based approach achieves 33\% better perceptual quality than the autoregressive baseline (0.283 vs 0.422 LPIPS). We validate our approach across three event camera datasets (BS-ERGB, HS-ERGB far/close) at varying sequence lengths (32-128 frames), demonstrating robust cross-dataset generalization with strong performance on unseen capture configurations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
