MuVAP: 面向野外对话轮次预测的多模态多方语音活动投影
MuVAP: Multimodal Multiparty Voice Activity Projection for Turn-taking Prediction in the Wild
AI总结：
提出MuVAP框架，通过将声学预测锚定到面部轨迹，实现从单声道音频和单摄像头视角进行说话人感知的轮次预测，并引入角色相对投影和AVCC数据集解决多方建模和因果跟踪问题。
摘要：
当前的多方对话轮次模型通常依赖于复杂的麦克风阵列或多摄像头设置，限制了它们在人与机器人交互场景中的适用性。我们提出了MuVAP，这是一个因果多模态框架，通过将声学预测锚定到面部轨迹来扩展语音活动投影，从而能够从单声道音频流和单摄像头视角进行说话人感知的轮次预测。为了解决建模多个说话人的组合复杂性，我们提出了角色相对投影，它将任意N说话人交互映射到一个固定的当前与下一个话语持有者状态。由于现有的视听数据集包含破坏因果跟踪的剪辑切换，我们引入了视听对话语料库，这是一个31小时的未剪辑、单摄像头多方对话数据集。评估表明，MuVAP在两人和三人场景下的转换-保持和下一说话人预测任务中优于强基线。
英文摘要：
Current multiparty turn-taking models often rely on complex microphone arrays or multi-camera setups, limiting their applicability in human-robot interaction scenarios. We introduce MuVAP, a causal multimodal framework that extends Voice Activity Projection by grounding acoustic predictions in face tracks, enabling speaker-aware turn-taking predictions from a monaural audio stream and a single camera view. To address the combinatorial complexity of modeling multiple speakers, we propose Role-Relative Projection, which maps any N-speaker interaction onto a fixed current versus next floor-holder state. Because existing audiovisual datasets contain disruptive editing cuts that break causal tracking, we introduce the Audio-Visual Conversation Corpus, a 31-hour dataset of unedited, single-camera multiparty conversations. Evaluations demonstrate that MuVAP outperforms strong baselines on Shift-Hold and next-speaker prediction tasks across two- and three-speaker settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
