VideoFDB: 评估对话代理中的全双工视觉-语音能力
VideoFDB: Evaluating Full-Duplex Vision-Speech Capabilities in Conversational Agents
AI总结：
提出首个全双工视听到视听（AV2AV）对话基准VideoFDB，通过237个真实视频片段、感知与生成行为分类以及基于评分规则的LM评判框架，系统评估代理在非语言对话动态中的表现，发现现有系统存在字幕崩溃和视觉流忽视等缺陷。
摘要：
自然的人类对话是全双工且视听融合的：人们同时说话和倾听，同时持续解读并产生非语言线索，如点头、微笑和手势。为了支持成功的人机交互，代理必须建模全双工视听对话；然而，现有的全双工基准仅评估语音。在这项工作中，我们提出了VideoFDB，这是首个评估全双工视听到视听（AV2AV）对话代理的基准。VideoFDB贡献了：(i) 237个来自真实世界视频通话的二元片段，涵盖11种非语言对话动态；(ii) 将感知行为与生成行为分离的分类法；(iii) 基于评分规则的LM评判评估框架，具有可解释的轴，用于评估关于非语言对话动态的对话质量。在开源和闭源的视觉-语音代理中，我们发现了系统性的失败模式：字幕崩溃和视觉流忽视，并且我们表明当前系统利用视觉进行显式视觉问答，但不用于自然对话中所需的流式联合视听基础。我们进一步评估了级联的语音到虚拟形象系统，发现其架构从根本上排除了全双工非语言线索的产生。作为全双工AV2AV交互的首个基准，VideoFDB为系统评估奠定了基础，我们希望这将加速下一代多模态对话代理的进步和发展。
英文摘要：
Natural human conversation is full-duplex and audio-visual: people simultaneously speak and listen while continuously interpreting and producing nonverbal cues, such as nods, smiles, and gestures. To support successful human-agent interaction, agents must model full-duplex audiovisual conversation; however, existing full-duplex benchmarks evaluate only speech. In this work, we present VideoFDB, the first benchmark to evaluate full-duplex audio-visual-to-audio-visual (AV2AV) conversational agents. VideoFDB contributes (i) 237 dyadic clips spanning 11 nonverbal conversational dynamics from real-world video calls, (ii) a taxonomy separating perception from generation behaviors, and (iii) a rubric-based LM-as-judge evaluation framework with interpretable axes for assessing conversational quality with respect to nonverbal conversational dynamics. Across open- and closed-source vision-speech agents, we find systematic failure modes: captioning collapse and visual-stream ignorance, and we show that current systems exploit vision for explicit visual question answering but not for the streaming joint audiovisual grounding required in natural conversation. We further evaluate cascaded speech-to-avatar systems and find that their architecture fundamentally precludes the production of full-duplex nonverbal cues. As the first benchmark for full-duplex AV2AV interaction, VideoFDB establishes a foundation for systematic evaluation and, we hope, will accelerate the advancement and development of next-generation multimodal conversational agents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
