AVI-Talking：学习音视频指令以实现富有表现力的3D说话人脸生成
AVI-Talking: Learning Audio-Visual Instructions for Expressive 3D Talking Face Generation
AI总结：
本文提出AVI-Talking系统，通过大语言模型理解语音并生成表情指令，再由扩散生成网络执行，实现与说话状态一致的富有表现力的3D说话人脸生成。
摘要：
尽管在3D语音驱动说话人脸生成中实现精确的唇音同步已取得显著进展，但合成与说话者说话状态一致的富有表现力的面部细节仍然具有挑战性。我们的目标是直接利用人类语音中固有的风格信息，生成与说话状态一致的富有表现力的说话人脸。本文提出AVI-Talking，一个用于富有表现力的说话人脸生成的音视频指令系统。该系统利用大语言模型（LLMs）提供的鲁棒上下文推理和幻觉能力，指导3D说话人脸的真实感合成。我们的两阶段策略不是直接从人类语音中学习面部运动，而是首先让LLMs理解音频信息并生成指令，这些指令隐含与语音无缝对应的富有表现力的面部细节。随后，基于扩散的生成网络执行这些指令。这一两阶段过程与LLMs的结合增强了模型的可解释性，并为用户提供了理解指令以及指定所需操作或修改的灵活性。大量实验证明了我们的方法在生成具有富有表现力的面部运动和一致情绪状态的生动说话人脸方面的有效性。
英文摘要：
While considerable progress has been made in achieving accurate lip synchronization for 3D speech-driven talking face generation, the task of incorporating expressive facial detail synthesis aligned with the speaker's speaking status remains challenging. Our goal is to directly leverage the inherent style information conveyed by human speech for generating an expressive talking face that aligns with the speaking status. In this paper, we propose AVI-Talking, an Audio-Visual Instruction system for expressive Talking face generation. This system harnesses the robust contextual reasoning and hallucination capability offered by Large Language Models (LLMs) to instruct the realistic synthesis of 3D talking faces. Instead of directly learning facial movements from human speech, our two-stage strategy involves the LLMs first comprehending audio information and generating instructions implying expressive facial details seamlessly corresponding to the speech. Subsequently, a diffusion-based generative network executes these instructions. This two-stage process, coupled with the incorporation of LLMs, enhances model interpretability and provides users with flexibility to comprehend instructions and specify desired operations or modifications. Extensive experiments showcase the effectiveness of our approach in producing vivid talking faces with expressive facial movements and consistent emotional status.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
