将面部生成集成到全双工口语对话系统中
Integrating Facial Generation into Full-Duplex Spoken Dialogue Systems
AI总结：
提出Moshi-Face，首个联合处理用户音频和面部输入并同时生成语音和面部运动的全双工对话模型，通过VQ-VAE面部编解码器和非自回归面部变换器实现实时视听对齐。
摘要：
全双工口语对话模型（如Moshi）能够实现自然、低延迟的语音对话。然而，它们仍局限于音频模态，缺乏人类交流中不可或缺的面部表情。我们提出Moshi-Face，这是首个全双工对话模型，能够联合处理用户的音频和面部输入，同时生成语音和面部运动。我们首先构建一个向量量化变分自编码器（VQ-VAE）作为面部编解码器，将从面部视频中提取的3D头部网格编码为紧凑的离散令牌（称为面部令牌），并反过来从这些令牌重建3D网格。然后，我们通过一个面部变换器模块扩展Moshi，该模块以非自回归方式生成面部令牌，使Moshi-Face能够实时生成同步的音频和面部令牌。实验表明，Moshi-Face在保持原始纯音频模型对话质量的同时，实现了低延迟的视听对齐。
英文摘要：
Full-duplex spoken dialogue models, such as Moshi, enable natural, low-latency voice conversations. However, they remain limited to the audio modality, lacking the facial expressions that are integral to human communication. We present Moshi-Face, the first full-duplex dialogue model that jointly processes the user's audio and facial input while simultaneously generating speech and facial motion. We first construct a vector-quantized variational autoencoder (VQ-VAE) as a face codec that encodes 3D head meshes extracted from facial videos into compact discrete tokens, referred to as face tokens, and conversely reconstructs 3D meshes from these tokens. We then extend Moshi with a Face Transformer module that generates face tokens non-autoregressively, enabling Moshi-Face to produce synchronized audio and face tokens in real time. Experiments show that Moshi-Face achieves audiovisual alignment at low latency while preserving the dialogue quality of the original audio-only model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
