arXiv 2411.036282024-11-06cs.CVcs.AI

StreamingBench：评估多模态大语言模型实现流式视频理解的差距

StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding

Tsinghua University（清华大学）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；Beijing University of Posts and Telecommunications（北京邮电大学）

Junming Lin, Zheng Fang, Chi Chen, Zihao Wan, Fuwen Luo, Peng Li, Yang Liu, Maosong Sun

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出StreamingBench，首个全面评估多模态大语言模型流式视频理解能力的基准，包含18个任务、900个视频和4,500个问答对，实验表明现有最先进模型（如Gemini 1.5 Pro和GPT-4o）仍显著低于人类水平。

摘要：

多模态大语言模型（MLLMs）的快速发展将其能力从图像理解扩展到了视频理解。然而，这些MLLMs大多主要关注离线视频理解，需要在任何查询之前对所有视频帧进行大量处理。这与人类实时观看、聆听、思考和响应流式输入的能力相比存在显著差距，凸显了当前MLLMs的局限性。在本文中，我们引入了StreamingBench，这是第一个旨在评估MLLMs流式视频理解能力的全面基准。StreamingBench评估流式视频理解的三个核心方面：（1）实时视觉理解，（2）全源理解，以及（3）上下文理解。该基准包含18个任务，涵盖900个视频和4,500个人工策划的问答对。每个视频包含在不同时间点提出的五个问题，以模拟连续的流式场景。我们在StreamingBench上对13个开源和专有MLLMs进行了实验，发现即使是最先进的专有MLLMs，如Gemini 1.5 Pro和GPT-4o，其表现也显著低于人类水平的流式视频理解能力。我们希望我们的工作能够促进MLLMs的进一步进步，使其在更真实的场景中接近人类水平的视频理解和交互。

英文摘要：

The rapid development of Multimodal Large Language Models (MLLMs) has expanded their capabilities from image comprehension to video understanding. However, most of these MLLMs focus primarily on offline video comprehension, necessitating extensive processing of all video frames before any queries can be made. This presents a significant gap compared to the human ability to watch, listen, think, and respond to streaming inputs in real time, highlighting the limitations of current MLLMs. In this paper, we introduce StreamingBench, the first comprehensive benchmark designed to evaluate the streaming video understanding capabilities of MLLMs. StreamingBench assesses three core aspects of streaming video understanding: (1) real-time visual understanding, (2) omni-source understanding, and (3) contextual understanding. The benchmark consists of 18 tasks, featuring 900 videos and 4,500 human-curated QA pairs. Each video features five questions presented at different time points to simulate a continuous streaming scenario. We conduct experiments on StreamingBench with 13 open-source and proprietary MLLMs and find that even the most advanced proprietary MLLMs like Gemini 1.5 Pro and GPT-4o perform significantly below human-level streaming video understanding capabilities. We hope our work can facilitate further advancements for MLLMs, empowering them to approach human-level video comprehension and interaction in more realistic scenarios.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。