发表机构
EPFL; Yonsei University（洛桑联邦理工学院; 延世大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
提出事件引导的神经视频编解码器（ENVC），利用共享事件流改进RGB压缩，在六个基准上相比DCMVC平均节省39.13%（PSNR-RGB）和67.63%（LPIPS）的BD率。
AI 中文摘要
神经视频编解码器主要从RGB帧中提取运动和时域上下文，为互补时域观测的跨模态引导留下了空间。事件流可以通过记录帧间的亮度变化来提供此类观测。在这项工作中，我们提出了一种事件引导的神经视频编解码器（ENVC），它利用编码器和解码器共享的事件来提高RGB压缩效率。对于运动编码，ENVC形成事件引导的运动先验，并编码剩余的运动残差。对于帧编码，事件条件预测器为门控时域上下文细化提供多尺度特征。为了支持在标准视频数据集上的训练和评估，我们合成了配对的RGB-事件数据，并通过与真实事件的比较来评估其预测效用。在六个基准测试中，相对于DCMVC，ENVC在使用PSNR-RGB时平均实现了39.13%的BD率节省，在使用LPIPS时平均实现了67.63%的BD率节省。进一步的分析表明，我们的增益在大运动序列上持续存在，且ENVC有效地学习了整合事件信息。这些结果证明了事件作为互补模态在降低RGB编码率方面的潜力。我们的模型和代码可在该https URL获得。
英文摘要
Neural video codecs derive motion and temporal contexts mainly from RGB frames, leaving room for cross-modal guidance from complementary temporal observations. Event streams can provide such observations by recording brightness changes between frames. In this work, we propose an Event-guided Neural Video Codec (ENVC) that uses events shared by the encoder and decoder to improve RGB compression efficiency. For motion coding, ENVC forms an event-guided motion prior and codes the remaining motion residual. For frame coding, an event-conditioned predictor supplies multi-scale features for gated temporal context refinement. To support training and evaluation on standard video datasets, we synthesize paired RGB-event data and assess its predictive utility through comparisons with real events. Across six benchmarks, ENVC achieves average BD-rate savings of 39.13% using PSNR-RGB and 67.63% using LPIPS relative to DCMVC. Further analyses show that our gains persist on large-motion sequences and that ENVC effectively learns to integrate event information. These results demonstrate the potential of events as a complementary modality for reducing the RGB coding rate. Our model and code are available at https://github.com/kjungwoo03/ENVC.
Comments28 pages. 21 figures
