arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2307.02090cs.CV

交互式对话头像生成

Interactive Conversational Head Generation

Mohan Zhou, Yalong Bai, Wei Zhang, Ting Yao, Tiejun Zhao

备注：
arXiv admin note: text overlap with arXiv:2112.13548
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出对话头像生成基准ViCo和ViCo-X，定义响应式倾听、表现力说话和对话头像生成三个任务，并给出基线方法，实现与真人协作完成多轮对话的数字人合成。

摘要：

我们引入了一个新的对话头像生成基准，用于合成面对面对话中单个对话者的行为。自动合成能够参与长时多轮对话的对话者的能力至关重要，并为数字人、虚拟代理和社交机器人等多种应用带来益处。现有研究主要关注说话头像生成（单向交互），由于缺少倾听和交互部分，这阻碍了为对话（双向）交互创建数字人的能力。在这项工作中，我们构建了两个数据集来解决这一问题：“ViCo”用于句子级别的独立说话和倾听头像生成任务，“ViCo-X”用于在多轮对话场景中合成对话者。基于ViCo和ViCo-X，我们定义了三个针对面对面对话中交互建模的新任务：1）响应式倾听头像生成，使倾听者通过非语言信号主动回应说话者；2）表现力说话头像生成，引导说话者感知倾听者的行为；3）对话头像生成，将说话/倾听能力整合到一个对话者中。除了数据集之外，我们还为上述三个任务提出了相应的基线解决方案。实验结果表明，我们的基线方法能够生成响应灵敏且生动的代理，它们可以与真人协作完成整个对话。项目页面：https://vico.solutions/。

英文摘要：

We introduce a new conversation head generation benchmark for synthesizing behaviors of a single interlocutor in a face-to-face conversation. The capability to automatically synthesize interlocutors which can participate in long and multi-turn conversations is vital and offer benefits for various applications, including digital humans, virtual agents, and social robots. While existing research primarily focuses on talking head generation (one-way interaction), hindering the ability to create a digital human for conversation (two-way) interaction due to the absence of listening and interaction parts. In this work, we construct two datasets to address this issue, ``ViCo'' for independent talking and listening head generation tasks at the sentence level, and ``ViCo-X'', for synthesizing interlocutors in multi-turn conversational scenarios. Based on ViCo and ViCo-X, we define three novel tasks targeting the interaction modeling during the face-to-face conversation: 1) responsive listening head generation making listeners respond actively to the speaker with non-verbal signals, 2) expressive talking head generation guiding speakers to be aware of listeners' behaviors, and 3) conversational head generation to integrate the talking/listening ability in one interlocutor. Along with the datasets, we also propose corresponding baseline solutions to the three aforementioned tasks. Experimental results show that our baseline method could generate responsive and vivid agents that can collaborate with real person to fulfil the whole conversation. Project page: https://vico.solutions/.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑