arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2507.09313cs.CV
收藏

ProactiveVideoQA:一个评估视频大型语言模型中主动交互的综合基准

ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Models

Wangxuan Institute of Computer Technology, Peking University(北京大学计算机技术研究院);Huawei Noah’s Ark Lab(华为诺亚实验室);School of Intelligence Science and Technology, University of Science and Technology Beijing(北京科技大学智能科学与技术学院);National Key Laboratory of General Artificial Intelligence(通用人工智能国家重点实验室)

Yueqian Wang, Xiaojun Meng, Yifan Wang, Huishuai Zhang, Dongyan Zhao

AI总结:

本文提出ProactiveVideoQA基准,用于评估视频大型语言模型的主动交互能力,并设计了考虑时间动态的新指标PAUC,实验证明其比传统指标更符合人类偏好。

摘要:

随着对多模态对话系统的研究日益深入,主动交互能力正逐渐受到认可。作为传统逐轮对话的替代方案,用户越来越期望多模态系统更具主动性,例如在视频播放过程中自主实时决定多轮响应的时机。为推动这一新兴领域的发展,我们推出了ProactiveVideoQA,这是首个评估系统主动交互能力的综合基准。由于模型响应在不同时间戳生成,我们进一步提出了PAUC,这是首个考虑模型响应时间动态变化的指标。这使得对在主动设置下运行的系统进行更准确的评估成为可能。通过对ProactiveVideoQA上各种基线系统进行广泛基准测试,以及对人类偏好进行用户研究,我们表明PAUC比通常仅考虑响应文本内容的传统评估指标更符合人类偏好。这些发现证明,PAUC在主动交互场景下能更忠实地评估用户体验。项目主页:https://github.com/yellow-binary-tree/ProactiveVideoQA

英文摘要:

With the growing research focus on multimodal dialogue systems, the capability for proactive interaction is gradually gaining recognition. As an alternative to conventional turn-by-turn dialogue, users increasingly expect multimodal systems to be more initiative, for example, by autonomously determining the timing of multi-turn responses in real time during video playback. To facilitate progress in this emerging area, we introduce ProactiveVideoQA, the first comprehensive benchmark to evaluate a system's ability to engage in proactive interaction. Since model responses are generated at varying timestamps, we further propose PAUC, the first metric that accounts for the temporal dynamics of model responses. This enables a more accurate evaluation of systems operating in proactive settings. Through extensive benchmarking of various baseline systems on ProactiveVideoQA and a user study of human preferences, we show that PAUC is in better agreement with human preferences than traditional evaluation metrics, which typically only consider the textual content of responses. These findings demonstrate that PAUC provides a more faithful assessment of user experience in proactive interaction scenarios. Project homepage: https://github.com/yellow-binary-tree/ProactiveVideoQA

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑