AV-Dialog：具有视听输入的口语对话模型
AV-Dialog: Spoken Dialogue Models with Audio-Visual Input
AI总结：
针对对话模型在嘈杂多说话者环境中的缺陷，提出首个多模态对话框架AV-Dialog，结合视听线索跟踪说话者、预测话轮转换并生成回复，通过多任务多阶段训练提升性能，为真实环境口语对话代理奠基。
摘要：
对话模型在嘈杂、多说话者环境中表现不佳，常产生无关回复和尴尬的话轮转换。我们提出AV-Dialog，这是首个利用音频和视觉线索跟踪目标说话者、预测话轮转换并生成连贯回复的多模态对话框架。通过将声学标记化与针对单语、合成及真实视听对话数据集的多任务、多阶段训练相结合，AV-Dialog实现了稳健的流式转录、语义接地的话轮边界检测和准确回复，从而形成自然的对话流。实验表明，AV-Dialog在干扰下优于仅音频模型，减少了转录错误，改进了话轮转换预测，并提升了人类评分的对话质量。这些结果凸显了“看”与“听”在说话者感知交互中的作用，为在真实世界嘈杂环境中稳健运行的口语对话代理铺平了道路。
英文摘要：
Dialogue models falter in noisy, multi-speaker environments, often producing irrelevant responses and awkward turn-taking. We present AV-Dialog, the first multimodal dialog framework that uses both audio and visual cues to track the target speaker, predict turn-taking, and generate coherent responses. By combining acoustic tokenization with multi-task, multi-stage training on monadic, synthetic, and real audio-visual dialogue datasets, AV-Dialog achieves robust streaming transcription, semantically grounded turn-boundary detection and accurate responses, resulting in a natural conversational flow. Experiments show that AV-Dialog outperforms audio-only models under interference, reducing transcription errors, improving turn-taking prediction, and enhancing human-rated dialogue quality. These results highlight the power of seeing as well as hearing for speaker-aware interaction, paving the way for {spoken} dialogue agents that perform {robustly} in real-world, noisy environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
