ENCORE：用于学习型视频压缩的事件辅助互补运动细化

中文摘要 AI 辅助

学习型视频压缩依赖精准的时间建模来消除相邻帧间的冗余。然而，现有大多数编解码器仅从离散采样的RGB帧中推断运动，其估计结果易受快速运动、模糊、遮挡、弱纹理、低光照及亮度突变的影响。事件相机异步捕捉RGB时间戳间的细粒度强度变化，因此能提供帧间动态的互补证据。本文提出ENCORE——一种用于学习型视频压缩的事件辅助互补运动细化框架。ENCORE首先采用互补运动表示（CMR）将对齐的RGB-事件特征分解为通用运动表示与模态特定运动表示；随后空间能量与冗余感知校准（SERIC）识别出相对于RGB活跃且新颖的事件特定响应，抑制弱或冗余证据，并预测候选流修正量；最后能量感知路由（EAR）确定该修正量应在何处以及以多大强度细化RGB流。事件仅作为运动建模的辅助模态，而RGB仍是唯一的编码与重建目标。在BS-ERGB、HQ-EVFI和CED数据集上的实验表明，该方法在不同数据集和GOP长度下均实现了一致的性能提升：在BS-ERGB上，ENCORE实现了最高20.80%的PSNR-RGB和22.14%的MS-SSIM-RGB BD-rate节省，在另外两个数据集上也保留了明显的改进效果。

英文摘要

Learned video compression relies on accurate temporal modeling to remove redundancy between adjacent frames. However, most existing codecs infer motion solely from discretely sampled RGB frames, making their estimates vulnerable to fast motion, blur, occlusion, weak texture, low illumination, and abrupt brightness changes. Event cameras asynchronously capture fine-grained intensity changes between RGB timestamps and therefore provide complementary evidence about inter-frame dynamics. We propose ENCORE, an Event-Assisted Complementary Motion Refinement framework for learned video compression. ENCORE first employs Complementary Motion Representation (CMR) to decompose aligned RGB-event features into common and modality-specific motion representations. Spatial Energy and Redundancy-Informed Calibration (SERIC) then identifies event-specific responses that are active and novel relative to RGB, suppresses weak or redundant evidence, and predicts a candidate flow correction. Finally, Energy-Aware Routing (EAR) determines where and how strongly the correction should refine the RGB flow. Events serve solely as an auxiliary modality for motion modeling, while RGB remains the only coding and reconstruction target. Experiments on BS-ERGB, HQ-EVFI, and CED demonstrate consistent gains across datasets and GOP lengths. On BS-ERGB, ENCORE achieves up to 20.80% PSNR-RGB and 22.14% MS-SSIM-RGB BD-rate savings, while retaining clear improvements on the other two datasets.

发表机构

• School of Computer Science, Wuhan University（武汉大学计算机学院）
• School of Artificial Intelligence, Ningbo University（宁波大学人工智能学院）
• National University of Singapore（新加坡国立大学）
• Zhejiang University of Technology（浙江工业大学）
• School of Information Engineering, Zhongnan University of Economics and Law（中南财经政法大学信息工程学院）

机构由 AI 辅助整理，请以论文原文为准。