让我们真实对话：面向面对面交谈的口语对话模型
Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversation
AI总结：
本文提出首个不依赖中间文本的面对面口语对话模型，并发布包含约340小时、约9,000段对话的大规模多模态语料库MultiDialog，通过语音-文本联合预训练适配大语言模型，实验验证了其在面对面交谈中的有效性。
摘要：
本文提出了一种新颖的面对面口语对话模型。该模型处理用户输入的视听语音并生成视听语音作为响应，标志着在不依赖中间文本的情况下创建虚拟化身聊天机器人系统的第一步。为此，我们新引入了MultiDialog，这是第一个大规模多模态（即音频和视觉）口语对话语料库，包含约340小时、约9,000段对话，基于开放域对话数据集TopicalChat录制。MultiDialog包含对话伙伴按照给定脚本表演的并行视听录音，并带有情感标注，我们期望这将为多模态合成开辟研究机会。我们的面对面口语对话模型融合了经过文本预训练的大语言模型，并通过引入语音-文本联合预训练将其适配到视听口语对话领域。通过大量实验，我们验证了该模型在促进面对面交谈方面的有效性。演示和数据分别可在https://multidialog.github.io和https://huggingface.co/datasets/IVLLab/MultiDialog获取。
英文摘要：
In this paper, we introduce a novel Face-to-Face spoken dialogue model. It processes audio-visual speech from user input and generates audio-visual speech as the response, marking the initial step towards creating an avatar chatbot system without relying on intermediate text. To this end, we newly introduce MultiDialog, the first large-scale multimodal (i.e., audio and visual) spoken dialogue corpus containing 340 hours of approximately 9,000 dialogues, recorded based on the open domain dialogue dataset, TopicalChat. The MultiDialog contains parallel audio-visual recordings of conversation partners acting according to the given script with emotion annotations, which we expect to open up research opportunities in multimodal synthesis. Our Face-to-Face spoken dialogue model incorporates a textually pretrained large language model and adapts it into the audio-visual spoken dialogue domain by incorporating speech-text joint pretraining. Through extensive experiments, we validate the effectiveness of our model in facilitating a face-to-face conversation. Demo and data are available at https://multidialog.github.io and https://huggingface.co/datasets/IVLLab/MultiDialog, respectively.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
