比特率受控扩散用于视频中运动与内容的解耦
Bitrate-Controlled Diffusion for Disentangling Motion and Content in Video
AI总结：
提出一种自监督框架，利用Transformer和低比特率向量量化解耦视频运动与内容，并以扩散模型进行条件生成，在说话头视频和卡通精灵上验证了有效性。
摘要：
我们提出了一种新颖且通用的框架，用于将视频数据解耦为动态运动成分和静态内容成分。我们的方法是一个自监督流程，相比以往工作具有更少的假设和归纳偏置：它利用基于Transformer的架构联合生成帧级运动和片段级内容的灵活隐式特征，并引入低比特率向量量化作为信息瓶颈以促进解耦并形成有意义的离散运动空间。比特率受控的潜在运动与内容被用作去噪扩散模型的条件输入，以促进自监督表示学习。我们在真实世界说话头视频上，通过运动迁移和自回归运动生成任务验证了我们的解耦表示学习框架。此外，我们还展示了我们的方法可以推广到其他类型的视频数据，例如2D卡通角色的像素精灵。我们的工作为自监督学习解耦视频表示提供了新视角，为视频分析和生成这一更广泛的领域做出了贡献。
英文摘要：
We propose a novel and general framework to disentangle video data into its dynamic motion and static content components. Our proposed method is a self-supervised pipeline with less assumptions and inductive biases than previous works: it utilizes a transformer-based architecture to jointly generate flexible implicit features for frame-wise motion and clip-wise content, and incorporates a low-bitrate vector quantization as an information bottleneck to promote disentanglement and form a meaningful discrete motion space. The bitrate-controlled latent motion and content are used as conditional inputs to a denoising diffusion model to facilitate self-supervised representation learning. We validate our disentangled representation learning framework on real-world talking head videos with motion transfer and auto-regressive motion generation tasks. Furthermore, we also show that our method can generalize to other types of video data, such as pixel sprites of 2D cartoon characters. Our work presents a new perspective on self-supervised learning of disentangled video representations, contributing to the broader field of video analysis and generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
