用眼睛倾听：跨时空的自体视觉共指基准测试
Listening with the Eyes: Benchmarking Egocentric Co-Speech Grounding across Space and Time
AI总结：
本文提出EcoG-Bench，通过严格测试语音-手势绑定，揭示多模态接口可能限制时间对齐线索的可观察性。
摘要：
在情境化协作中，说话者常常使用故意模糊的指示性命令（例如，'给我那'），其指称只有通过将语音与短暂的共语指向笔画对齐才能确定。然而，许多具身体验基准测试允许语言-only的捷径，使MLLMs能够在不学习所需的音频-视觉对齐的情况下表现良好。为弥合这一差距，我们引入了自体视觉共指（Egocentric Co-Speech Grounding, EcoG），其中共指只有在代理联合预测'什么'、'哪里'和'什么时候'时才能执行。为了操作化这一概念，我们提出了EcoG-Bench，一个仅用于评估的双语（EN/ZH）诊断基准测试，包含811个自体视频片段，具有密集的空间注释和毫秒级笔画监督。它在渐进认知评估协议下组织。对最先进的MLLMs进行基准测试揭示了严重的可执行性差距：尽管人类受试者在EcoG-Bench上几乎达到满分表现（96.9%的严格Eco-Accuracy），但最佳的原生视频-音频设置仍较低（Gemini-3-Pro：17.0%）。此外，在诊断消融中，将原生视频-音频接口替换为带时间戳的帧样本和外部验证的ASR（带有词级时间）显著提高了相同模型的性能（17.0%→42.9%）。总体而言，EcoG-Bench为事件级的语音-手势绑定提供了严格且可执行的测试平台，并表明多模态接口可能独立于模型推理能力，会限制时间对齐线索的可观察性。
英文摘要：
In situated collaboration, speakers often use intentionally underspecified deictic commands (e.g., ``pass me \textit{that}''), whose referent becomes identifiable only by aligning speech with a brief co-speech pointing \emph{stroke}. However, many embodied benchmarks admit language-only shortcuts, allowing MLLMs to perform well without learning the \emph{audio--visual alignment} required by deictic interaction. To bridge this gap, we introduce \textbf{Egocentric Co-Speech Grounding (EcoG)}, where grounding is executable only if an agent jointly predicts \textit{What}, \textit{Where}, and \textit{When}. To operationalize this, we present \textbf{EcoG-Bench}, an evaluation-only bilingual (EN/ZH) diagnostic benchmark of \textbf{811} egocentric clips with dense spatial annotations and millisecond-level stroke supervision. It is organized under a \textbf{Progressive Cognitive Evaluation} protocol. Benchmarking state-of-the-art MLLMs reveals a severe executability gap: while human subjects achieve near-ceiling performance on EcoG-Bench (\textbf{96.9\%} strict Eco-Accuracy), the best native video-audio setting remains low (Gemini-3-Pro: \textbf{17.0\%}). Moreover, in a diagnostic ablation, replacing the native video--audio interface with timestamped frame samples and externally verified ASR (with word-level timing) substantially improves the same model (\textbf{17.0\%}$\to$\textbf{42.9\%}). Overall, EcoG-Bench provides a strict, executable testbed for event-level speech--gesture binding, and suggests that multimodal interfaces may bottleneck the observability of temporal alignment cues, independently of model reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
