MoCha：迈向电影级说话角色合成
MoCha: Towards Movie-Grade Talking Character Synthesis
AI总结：
本文提出首个说话角色生成模型MoCha，通过语音-视频窗口注意力机制实现音画同步，利用联合训练策略提升泛化能力，并设计结构化提示模板首次实现多角色回合制对话，为AI电影叙事设定新标准。
摘要：
视频生成的最新进展在动作真实感方面取得了令人瞩目的成果，但往往忽略了角色驱动的叙事，这是自动化电影和动画生成的一项关键任务。我们引入了Talking Characters（说话角色），这是一个更贴近实际的任务，旨在直接从语音和文本生成说话角色的动画。与talking head（说话头部）不同，Talking Characters旨在生成超越面部区域的一个或多个角色的全身像。在本文中，我们提出了MoCha，这是首个用于生成说话角色的模型。为确保视频与语音的精确同步，我们提出了一种语音-视频窗口注意力机制，该机制能有效地对齐语音和视频token。为解决大规模语音标注视频数据集的稀缺问题，我们引入了一种联合训练策略，同时利用语音标注和文本标注的视频数据，显著提升了模型在多样化角色动作上的泛化能力。我们还设计了带有角色标签的结构化提示模板，首次实现了基于回合制对话的多角色对话——允许AI生成的角色参与具有电影级连贯性的上下文感知对话。广泛的定性和定量评估，包括人类偏好研究和基准比较，表明MoCha为AI生成的电影叙事设定了新标准，实现了卓越的真实感、表现力、可控性和泛化能力。
英文摘要：
Recent advancements in video generation have achieved impressive motion realism, yet they often overlook character-driven storytelling, a crucial task for automated film, animation generation. We introduce Talking Characters, a more realistic task to generate talking character animations directly from speech and text. Unlike talking head, Talking Characters aims at generating the full portrait of one or more characters beyond the facial region. In this paper, we propose MoCha, the first of its kind to generate talking characters. To ensure precise synchronization between video and speech, we propose a speech-video window attention mechanism that effectively aligns speech and video tokens. To address the scarcity of large-scale speech-labeled video datasets, we introduce a joint training strategy that leverages both speech-labeled and text-labeled video data, significantly improving generalization across diverse character actions. We also design structured prompt templates with character tags, enabling, for the first time, multi-character conversation with turn-based dialogue-allowing AI-generated characters to engage in context-aware conversations with cinematic coherence. Extensive qualitative and quantitative evaluations, including human preference studies and benchmark comparisons, demonstrate that MoCha sets a new standard for AI-generated cinematic storytelling, achieving superior realism, expressiveness, controllability and generalization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
