记忆是说话人脸生成中的一对多映射缓解器
Memories are One-to-Many Mapping Alleviators in Talking Face Generation
AI总结：
针对说话人脸生成中音频到视频的一对多映射歧义问题，本文提出 MemFace，通过隐式记忆补充音频到表情语义、显式记忆增强神经渲染细节，并在多场景实验中显著超越现有方法。
摘要：
说话人脸生成旨在生成由输入音频驱动的目标人物的照片级真实感视频肖像。由于其从输入音频到输出视频的一对多映射本质（例如，一段语音内容可能具有多种可行的视觉外观），像以往工作那样学习确定性映射会在训练过程中带来歧义，从而导致较差的视觉结果。尽管这种一对多映射可以通过两阶段框架（即先使用音频到表情模型，再使用神经渲染模型）得到部分缓解，但这仍然不够充分，因为预测是在没有足够信息（例如情绪、皱纹等）的情况下产生的。在本文中，我们提出 MemFace，通过分别遵循这两个阶段意义的隐式记忆和显式记忆来补充缺失信息。更具体地说，隐式记忆被用于音频到表情模型中，以捕捉音频-表情共享空间中的高级语义，而显式记忆被用于神经渲染模型中，以帮助合成像素级细节。我们的实验结果表明，所提出的 MemFace 在多种场景下均持续且显著地超过所有最先进结果。
英文摘要：
Talking face generation aims at generating photo-realistic video portraits of a target person driven by input audio. Due to its nature of one-to-many mapping from the input audio to the output video (e.g., one speech content may have multiple feasible visual appearances), learning a deterministic mapping like previous works brings ambiguity during training, and thus causes inferior visual results. Although this one-to-many mapping could be alleviated in part by a two-stage framework (i.e., an audio-to-expression model followed by a neural-rendering model), it is still insufficient since the prediction is produced without enough information (e.g., emotions, wrinkles, etc.). In this paper, we propose MemFace to complement the missing information with an implicit memory and an explicit memory that follow the sense of the two stages respectively. More specifically, the implicit memory is employed in the audio-to-expression model to capture high-level semantics in the audio-expression shared space, while the explicit memory is employed in the neural-rendering model to help synthesize pixel-level details. Our experimental results show that our proposed MemFace surpasses all the state-of-the-art results across multiple scenarios consistently and significantly.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
