MoCha:端到端视频人物替换无需结构引导
MoCha:End-to-End Video Character Replacement without Structural Guidance
AI总结：
MoCha提出了一种无需结构引导的端到端视频人物替换方法，通过单帧掩码实现高效替换，并设计了三种数据集以克服数据稀缺问题。
摘要：
可控的视频人物替换仍是一个具有挑战性的问题，因为缺乏配对的视频数据。先前的工作主要依赖于重建基于的范式，需要每帧分割掩码和显式的结构引导（例如骨骼、深度）。然而，这种依赖性严重限制了其在涉及遮挡、人物-物体交互、不寻常姿势或挑战性照明的复杂场景中的通用性，常常导致视觉伪影和时间不一致。在本文中，我们提出了MoCha，一种开创性的框架，通过仅需要一个任意帧掩码来绕过这些限制。为了有效适应多模态输入条件并增强面部身份，我们引入了条件感知的RoPE，并采用基于强化学习的后训练阶段。此外，为了克服合格配对训练数据的稀缺性，我们提出了一套全面的数据构建流程。具体来说，我们设计了三个专门的数据集：一个使用Unreal Engine 5（UE5）构建的高保真渲染数据集，一个通过当前肖像动画技术合成的表情驱动数据集，以及一个从现有视频-掩码对衍生出的增强数据集。广泛的实验表明，我们的方法在很大程度上优于现有最先进的方法。我们将发布代码以促进进一步研究。请参考我们的项目页面获取更多细节：orange-3dv-team.github.io/MoCha
英文摘要：
Controllable video character replacement with a user-provided identity remains a challenging problem due to the lack of paired video data. Prior works have predominantly relied on a reconstruction-based paradigm that requires per-frame segmentation masks and explicit structural guidance (e.g., skeleton, depth). This reliance, however, severely limits their generalizability in complex scenarios involving occlusions, character-object interactions, unusual poses, or challenging illumination, often leading to visual artifacts and temporal inconsistencies. In this paper, we propose MoCha, a pioneering framework that bypasses these limitations by requiring only a single arbitrary frame mask. To effectively adapt the multi-modal input condition and enhance facial identity, we introduce a condition-aware RoPE and employ an RL-based post-training stage. Furthermore, to overcome the scarcity of qualified paired-training data, we propose a comprehensive data construction pipeline. Specifically, we design three specialized datasets: a high-fidelity rendered dataset built with Unreal Engine 5 (UE5), an expression-driven dataset synthesized by current portrait animation techniques, and an augmented dataset derived from existing video-mask pairs. Extensive experiments demonstrate that our method substantially outperforms existing state-of-the-art approaches. We will release the code to facilitate further research. Please refer to our project page for more details: orange-3dv-team.github.io/MoCha
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
