基于对比学习的多参考生成式人脸视频压缩
Multi-Reference Generative Face Video Compression with Contrastive Learning
AI总结：
本文提出多参考深度动画编解码器MRDAC，通过多参考动画和对比学习缓解生成式人脸视频压缩中的重建漂移问题，以更少参考帧实现更长序列压缩和更高重建精度。
摘要：
生成式人脸视频编码（GFVC）已被证明是一种有潜力实现低延迟、低比特率视频会议的方法。与比特率低于10kbps的传统编解码器相比，GFVC框架在编码效率上实现了超过70%的比特率节省，获得了极大的增益。在近期MPEG/JVET标准化工作中，使用GFVC框架重建视频序列所需的全部信息已被采纳为现有压缩流程中补充增强信息（SEI）的一部分。鉴于这一进展，我们旨在解决先前GFVC框架中处理不足的一个挑战，即当参考帧与目标帧之间的距离增大时出现的重建漂移问题。这一挑战导致需要更频繁地更新参考缓冲区，即传输更多的Intra-refresh帧，而这些帧是GFVC比特流中开销最大的部分。为克服这一问题，我们提出以多参考动画作为一种鲁棒方法来最小化重建漂移，尤其是在双向预测模式下使用时效果更佳。此外，我们提出了一种用于多参考动画的对比学习公式。我们观察到，使用对比学习框架能够增强动画生成器的表示能力。由此得到的框架MRDAC（多参考深度动画编解码器）因此能够以更少的参考帧压缩更长的序列，或在与此前框架相当的比特率下实现重建精度的显著提升。定量和定性结果表明，与此前的GFVC方法相比，编码质量和重建质量均有显著提升，并且在存在大幅度姿态和面部表情变化的情况下，动画质量更加准确。
英文摘要：
Generative face video coding (GFVC) has been demonstrated as a potential approach to low-latency, low bitrate video conferencing. GFVC frameworks achieve an extreme gain in coding efficiency with over 70% bitrate savings when compared to conventional codecs at bitrates below 10kbps. In recent MPEG/JVET standardization efforts, all the information required to reconstruct video sequences using GFVC frameworks are adopted as part of the supplemental enhancement information (SEI) in existing compression pipelines. In light of this development, we aim to address a challenge that has been weakly addressed in prior GFVC frameworks, i.e., reconstruction drift as the distance between the reference and target frames increases. This challenge creates the need to update the reference buffer more frequently by transmitting more Intra-refresh frames, which are the most expensive element of the GFVC bitstream. To overcome this problem, we propose instead multiple reference animation as a robust approach to minimizing reconstruction drift, especially when used in a bi-directional prediction mode. Further, we propose a contrastive learning formulation for multi-reference animation. We observe that using a contrastive learning framework enhances the representation capabilities of the animation generator. The resulting framework, MRDAC (Multi-Reference Deep Animation Codec) can therefore be used to compress longer sequences with fewer reference frames or achieve a significant gain in reconstruction accuracy at comparable bitrates to previous frameworks. Quantitative and qualitative results show significant coding and reconstruction quality gains compared to previous GFVC methods, and more accurate animation quality in presence of large pose and facial expression changes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
