DialogueNeRF：迈向逼真的化身面对面对话视频生成
DialogueNeRF: Towards Realistic Avatar Face-to-Face Conversation Video Generation
AI总结：
本文提出基于 NeRF 的统一框架 DialogueNeRF，联合建模说话者与听者以生成逼真面对面对话视频，并构建含34段视频的数据集验证其真实感与个体风格保持能力。
摘要：
对话是元宇宙中虚拟化身活动的重要组成部分。随着自然语言处理的发展，文本和语音对话生成已经取得重大突破。然而，面对面对话在日常对话中占绝大多数，而现有方法大多聚焦于单人口语头部生成。在本工作中，我们更进一步，考虑生成逼真的面对面对话视频。对话生成比单人口语头部生成更具挑战性，因为它不仅需要生成照片级真实的个体口语头部，还要求听者对说话者作出回应。本文提出一种基于神经辐射场（NeRF）的新型统一框架来解决该任务。具体而言，我们使用一个 NeRF 框架同时建模说话者和听者，并采用不同条件控制个体表情。说话者由音频信号驱动，而听者的回应同时依赖视觉和声学信息。通过这种方式，可在人类化身之间生成面对面对话视频，并且所有对话者都在同一网络内建模。此外，为促进未来对该任务的研究，我们收集了一个包含34段视频的新的人类对话数据集。定量和定性实验从不同方面评估了我们的方法，例如图像质量、姿态序列趋势以及渲染视频的自然度。实验结果表明，所得视频中的化身能够进行逼真的对话，并保持个体风格。所有代码、数据和模型都将公开发布。
英文摘要：
Conversation is an essential component of virtual avatar activities in the metaverse. With the development of natural language processing, textual and vocal conversation generation has achieved a significant breakthrough. However, face-to-face conversations account for the vast majority of daily conversations, while most existing methods focused on single-person talking head generation. In this work, we take a step further and consider generating realistic face-to-face conversation videos. Conversation generation is more challenging than single-person talking head generation, since it not only requires generating photo-realistic individual talking heads but also demands the listener to respond to the speaker. In this paper, we propose a novel unified framework based on neural radiance field (NeRF) to address this task. Specifically, we model both the speaker and listener with a NeRF framework, with different conditions to control individual expressions. The speaker is driven by the audio signal, while the response of the listener depends on both visual and acoustic information. In this way, face-to-face conversation videos are generated between human avatars, with all the interlocutors modeled within the same network. Moreover, to facilitate future research on this task, we collect a new human conversation dataset containing 34 clips of videos. Quantitative and qualitative experiments evaluate our method in different aspects, e.g., image quality, pose sequence trend, and naturalness of the rendering videos. Experimental results demonstrate that the avatars in the resulting videos are able to perform a realistic conversation, and maintain individual styles. All the code, data, and models will be made publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
