InstructAvatar：用于头像生成的文本引导情感与运动控制
InstructAvatar: Text-Guided Emotion and Motion Control for Avatar Generation
AI总结：
本文提出InstructAvatar，一种文本引导的2D头像生成方法，通过自然语言控制情感和面部运动，利用自动标注和双分支扩散生成器，实现细粒度情感控制、高唇形同步和自然度，优于现有方法。
摘要：
近期，说话头像生成模型在实现与音频的逼真和准确唇形同步方面取得了进展，但在控制和传达头像的细微表情与情感方面往往不足，导致生成的视频不够生动和可控。在本文中，我们提出了一种新颖的文本引导方法，用于生成富有情感表达的2D头像，为最终视频提供细粒度控制、改进的交互性和泛化能力。我们的框架名为InstructAvatar，利用自然语言界面来控制头像的情感以及面部运动。技术上，我们设计了一个自动标注流程来构建指令-视频配对训练数据集，并配备了一个新颖的双分支基于扩散的生成器，以同时根据音频和文本指令预测头像。实验结果表明，InstructAvatar生成的结果与两种条件均高度对齐，并在细粒度情感控制、唇形同步质量和自然度方面优于现有方法。我们的项目页面是 https://wangyuchi369.github.io/InstructAvatar/。
英文摘要：
Recent talking avatar generation models have made strides in achieving realistic and accurate lip synchronization with the audio, but often fall short in controlling and conveying detailed expressions and emotions of the avatar, making the generated video less vivid and controllable. In this paper, we propose a novel text-guided approach for generating emotionally expressive 2D avatars, offering fine-grained control, improved interactivity, and generalizability to the resulting video. Our framework, named InstructAvatar, leverages a natural language interface to control the emotion as well as the facial motion of avatars. Technically, we design an automatic annotation pipeline to construct an instruction-video paired training dataset, equipped with a novel two-branch diffusion-based generator to predict avatars with audio and text instructions at the same time. Experimental results demonstrate that InstructAvatar produces results that align well with both conditions, and outperforms existing methods in fine-grained emotion control, lip-sync quality, and naturalness. Our project page is https://wangyuchi369.github.io/InstructAvatar/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
