PAI-Actor：动态场景中的电影级多角色替换

中文摘要 AI 辅助

我们提出了PAI-Actor，一个用于动态电影场景中角色替换的电影级多角色动画框架。与主要驱动单个静态图像或单个主体的传统动画系统不同，我们的目标是在真实视频片段中替换并动画化多个角色，同时保留原始场景动态、相机运动和背景内容。这一设定尤其具有挑战性，因为生成的角色必须在动作和交互上与源表演保持一致，同时在光照、阴影、构图和整体电影外观上与周围背景相匹配。为解决这一问题，我们将多角色动画表述为一个结构引导的人体恢复问题，并利用高质量电影数据构建了一个电影驱动的训练流程。此外，为支持实际电影制作，我们引入了一个双向到自回归的蒸馏框架：我们首先训练一个双向扩散变换器，用于在1080P分辨率下生成高质量短片，然后将其蒸馏为一个自回归的视频到视频模型，以实现高效推理和更长视频的生成。实验表明，PAI-Actor能够实现高保真度的多角色动画，具有强大的场景一致性、电影级视觉质量和高效的长序列生成能力。

英文摘要

We present PAI-Actor, a cinematic multi-character animation framework for character replacement in dynamic movie scenes. Unlike conventional animation systems that mainly drive a single static image or a single subject, our goal is to replace and animate multiple characters within real video clips while preserving the original scene dynamics, camera motion, and background content. This setting is particularly challenging because the generated characters must remain consistent with the source performance in motion and interaction, while also matching the surrounding background in lighting, shadow, composition, and overall cinematic appearance. To address this, we formulate multi-character animation as a structure-guided human recovery problem and build a movie-driven training pipeline from high-quality film data. Furthermore, to support practical cinematic production, we introduce a bidirectional-to-autoregressive distillation framework: we first train a bidirectional diffusion transformer for high-quality short-clip generation at 1080P resolution, and then distill it into an autoregressive video-to-video model for efficient inference and longer video generation. Experiments show that PAI-Actor enables high-fidelity multi-character animation with strong scene consistency, cinematic visual quality, and efficient long-form generation.

发表机构

• Utopai Studios（Utopai工作室）
• University of California, Irvine（加州大学尔湾分校）
• Nanyang Technological University（南洋理工大学）
• Show Lab, National University of Singapore（新加坡国立大学Show Lab）
• National University of Singapore（新加坡国立大学）

机构由 AI 辅助整理，请以论文原文为准。