Kling-Avatar：面向级联长时长虚拟人动画合成的多模态指令基础对齐
Kling-Avatar: Grounding Multimodal Instructions for Cascaded Long-Duration Avatar Animation Synthesis
AI总结：
针对现有音频驱动虚拟人缺乏指令交流目的建模的问题，提出Kling-Avatar级联框架，通过MLLM导演生成蓝图视频并并行合成子片段，实现了高保真、长时长且语义可控的虚拟人动画生成。
摘要：
音频驱动虚拟人视频生成的最新进展显著提升了视听真实感。然而，现有方法仅将指令条件视为由声学或视觉线索驱动的低层次追踪，而未对指令传达的交流目的进行建模。这一局限损害了叙事连贯性和角色表现力。为弥补这一空白，我们引入了Kling-Avatar，一种将多模态指令理解与照片级真实肖像生成统一起来的新型级联框架。我们的方法采用两阶段流水线。在第一阶段，我们设计了一个多模态大语言模型（MLLM）导演，它根据多样的指令信号生成蓝图视频，从而控制角色动作和情绪等高层语义。在第二阶段，在蓝图关键帧的引导下，我们使用首尾帧策略并行生成多个子片段。这种从全局到局部的框架在保留细粒度细节的同时，忠实地编码了多模态指令背后的高层意图。我们的并行架构还能快速稳定地生成长时长视频，使其适用于数字人直播和视频博客等实际应用。为全面评估我们的方法，我们构建了一个包含375个精选样本的基准，涵盖多样的指令和具有挑战性的场景。大量实验表明，Kling-Avatar能够生成最高1080p和48 fps的生动、流畅的长时长视频，在唇形同步精度、情绪与动态表现力、指令可控性、身份保持性和跨域泛化能力方面取得了卓越的性能。这些结果确立了Kling-Avatar作为语义基础对齐、高保真音频驱动虚拟人合成的新基准。
英文摘要：
Recent advances in audio-driven avatar video generation have significantly enhanced audio-visual realism. However, existing methods treat instruction conditioning merely as low-level tracking driven by acoustic or visual cues, without modeling the communicative purpose conveyed by the instructions. This limitation compromises their narrative coherence and character expressiveness. To bridge this gap, we introduce Kling-Avatar, a novel cascaded framework that unifies multimodal instruction understanding with photorealistic portrait generation. Our approach adopts a two-stage pipeline. In the first stage, we design a multimodal large language model (MLLM) director that produces a blueprint video conditioned on diverse instruction signals, thereby governing high-level semantics such as character motion and emotions. In the second stage, guided by blueprint keyframes, we generate multiple sub-clips in parallel using a first-last frame strategy. This global-to-local framework preserves fine-grained details while faithfully encoding the high-level intent behind multimodal instructions. Our parallel architecture also enables fast and stable generation of long-duration videos, making it suitable for real-world applications such as digital human livestreaming and vlogging. To comprehensively evaluate our method, we construct a benchmark of 375 curated samples covering diverse instructions and challenging scenarios. Extensive experiments demonstrate that Kling-Avatar is capable of generating vivid, fluent, long-duration videos at up to 1080p and 48 fps, achieving superior performance in lip synchronization accuracy, emotion and dynamic expressiveness, instruction controllability, identity preservation, and cross-domain generalization. These results establish Kling-Avatar as a new benchmark for semantically grounded, high-fidelity audio-driven avatar synthesis.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
