OmniPro: 一个全面的多模态流视频理解基准测试
OmniPro: A Comprehensive Benchmark for Omni-Proactive Streaming Video Understanding
AI总结:
本文提出OmniPro基准测试,旨在评估多模态感知、主动响应和多样视频理解任务,通过2700个经人类验证的样本覆盖9个子任务和3个认知层次,揭示音频在流视频理解中的关键作用及模型在长时间任务中的鲁棒性问题。
摘要:
多模态流视频理解,即从连续音频视频流中自主决定何时说话和说什么,是多模态大语言模型的一种新兴能力。现有基准测试在三个方面存在不足:它们主要依赖视觉信号,采用轮询或固定时间戳协议而不是真正的主动评估,并且只涵盖有限的任务范围,从而无法可靠地评估和区分多模态流模型。我们提出了OmniPro,这是第一个联合评估多模态感知、主动响应和多样化视频理解任务的基准测试。它包含2700个经人类验证的样本,涵盖9个子任务和3个认知层次,覆盖6种基本视频理解能力。值得注意的是,84%的样本需要音频信号(语音或非语音),并且每个样本都带有模态隔离标签,以实现细粒度的多模态分析。我们进一步引入了一种双模式评估协议:探测模式通过在每个真实触发前后查询模型来评估内容理解,而在线模式通过要求模型在流输入中自主决定何时响应来评估全面的主动能力。评估11个代表性模型后发现三个关键发现:(1)音频提供了一致的收益,但不同模型的利用情况差异很大;(2)性能随时间显著下降,表明长期鲁棒性有限;(3)非语音音频感知仍然是最弱的维度。
英文摘要:
Omni-proactive streaming video understanding, i.e., autonomously deciding when to speak and what to say from continuous audio-visual streams, is an emerging capability of omni-modal large language models. Existing benchmarks fall short in three key aspects: they rely primarily on visual signals, adopt polling or fixed-timestamp protocols instead of true proactive evaluation, and cover only a limited range of tasks, preventing reliable assessment and differentiation of omni-proactive streaming models. We present OmniPro, the first benchmark to jointly evaluate omni-modal perception, proactive responding, and diverse video understanding tasks. It comprises 2,700 human-verified samples spanning 9 sub-tasks and 3 cognitive levels, covering 6 basic video understanding capabilities. Notably, 84% of samples require audio signals (speech or non-speech), and each sample is annotated with modality-isolation labels to enable fine-grained multimodal analysis. We further introduce a dual-mode evaluation protocol: Probe mode assesses content understanding by querying the model before and after each ground-truth trigger, while Online mode evaluates full proactive ability by requiring models to autonomously decide when to respond in streaming input. Evaluating 11 representative models reveals three key findings: (1) audio provides consistent gains but with highly variable utilization across models, (2) performance degrades significantly over time, indicating limited long-horizon robustness, and (3) non-speech audio perception remains the weakest dimension.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。