发表机构
Intel Corporation（英特尔公司）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
ReGenVC是首个结合超低位率编码与实时解码的端到端生成式视频编解码器，通过技术优化实现24 fps输出，内存占用显著降低。
AI 中文摘要
我们提出了ReGenVC，这是一种端到端生成式视频编解码器，可将人脸视频压缩至超低位率并实时解码。编码器将源视频片段压缩为紧凑比特流，包括神经压缩的首帧、每帧姿态关键点和元数据，77帧序列总大小约26 kB。解码器是四步蒸馏扩散Transformer，基于传输的姿态和参考帧重建视频。与x264/x265相比，ReGenVC将比特率降至传统编解码器的约十分之一（约26 kB对250-280 kB，即可实现基本无伪影的重建）；在匹配的超低位率下，传统编解码器会出现块效应伪影，而ReGenVC利用强大的生成先验保持清晰。部署此类编解码器的核心障碍是解码器延迟：Transformer和VAE组件的多步采样对于交互式使用过慢。我们通过四步蒸馏和三种保留模型的系统技术使解码器实现实时运行：（i）八GPU统一序列并行（Ulysses与Ring），（ii）空间拆分VAE，（iii）三级重叠流水线；分析时序模型表征了实时可行区域。在八GPU节点上，系统维持24 fps输出（25帧窗口耗时972 ms，处于1000 ms预算内），可实现无帧欠载的实时浏览器流。混合CPU-GPU部署进一步在CPU上以24 fps运行编码器，并将解码器侧的单次条件编码器卸载至CPU，将每GPU内存峰值从21.1 GB降至约7.7 GB。据我们所知，ReGenVC是首个在八GPU系统上结合超低位率编码与实时解码的端到端生成式视频编解码器。
英文摘要
We present ReGenVC, an end-to-end generative video codec that compresses talking-head video to an ultra-low bitrate and decodes it in real time. The encoder reduces a source clip to a compact bitstream -- a neurally compressed first frame, per-frame pose keypoints, and metadata -- totaling about 26 kB for a 77-frame sequence. The decoder is a four-step distilled diffusion transformer that reconstructs the video conditioned on the transmitted pose and reference frame. Compared with x264/x265, ReGenVC reduces the bitrate to roughly one tenth of that required by traditional codecs (about 26 kB vs. 250--280 kB for essentially artifact-free reconstruction); at a matched ultra-low bitrate, conventional codecs collapse into blocking artifacts while ReGenVC stays sharp by exploiting a strong generative prior. The central obstacle to deploying such a codec is decoder latency: multi-step sampling with transformer and VAE components is too slow for interactive use. We make the decoder real-time through four-step distillation and three model-preserving system techniques: (i) eight-GPU unified sequence parallelism (Ulysses & Ring), (ii) a spatially-split VAE, and (iii) a three-stage overlapped pipeline; an analytical timing model characterizes the real-time feasibility region. On an 8-GPU node, the system sustains 24 fps output (972 ms per 25-frame window, within the 1000 ms budget), enabling a live browser stream without observed frame underruns. A hybrid CPU-GPU deployment further runs the encoder on the CPU at 24 fps and offloads the decoder-side one-shot conditioning encoders to the CPU, reducing the per-GPU memory peak from 21.1 GB to about 7.7 GB. To our knowledge, ReGenVC is the first end-to-end generative video codec to combine ultra-low-bitrate encoding with real-time decoding on an 8-GPU system.
Comments12 pages, 5 figures, 5 tables
