用于快节奏视频理解的VideoAds
VideoAds for Fast-Paced Video Understanding
AI总结:
本文提出首个专为评估多模态大语言模型在广告视频上性能的基准数据集VideoAds,包含手动标注的视觉查找、视频摘要和视觉推理任务,实验表明开源模型Qwen2.5-VL-72B优于GPT-4o等专有模型,强调了提升时间建模能力的必要性。
摘要:
广告视频是目的驱动信息的丰富且宝贵的来源,包含旨在吸引观众的高质量视觉、文本和上下文线索。由于其结构化的叙事和快速的场景转换,它们通常比相同时长的普通视频更复杂,对多模态大语言模型(MLLMs)构成了重大挑战。在这项工作中,我们引入了VideoAds,这是第一个专为评估MLLMs在广告视频上的性能而定制的数据集。VideoAds包含经过精心策划的具有复杂时间结构的广告视频,并附带了跨三个核心任务:视觉查找、视频摘要和视觉推理的**手动**标注的多样化问题。我们提出了一种定量测量方法,以在视频复杂度方面将VideoAds与现有基准进行比较。通过大量实验,我们发现开源MLLM Qwen2.5-VL-72B在VideoAds上达到了73.35%的准确率,优于GPT-4o(66.82%)和Gemini-1.5 Pro(69.66%);这两个专有模型在视频摘要和推理方面尤其落后于开源模型,但在视觉查找方面表现最佳。值得注意的是,人类专家轻松达到了94.27%的惊人准确率。这些结果强调了提升MLLMs时间建模能力的必要性,并突出了VideoAds作为未来理解需要高FPS采样的视频研究的潜在关键基准。数据集和评估代码将在https://videoadsbenchmark.netlify.app公开发布。
英文摘要:
Advertisement videos serve as a rich and valuable source of purpose-driven information, encompassing high-quality visual, textual, and contextual cues designed to engage viewers. They are often more complex than general videos of similar duration due to their structured narratives and rapid scene transitions, posing significant challenges to multi-modal large language models (MLLMs). In this work, we introduce VideoAds, the first dataset tailored for benchmarking the performance of MLLMs on advertisement videos. VideoAds comprises well-curated advertisement videos with complex temporal structures, accompanied by \textbf{manually} annotated diverse questions across three core tasks: visual finding, video summary, and visual reasoning. We propose a quantitative measure to compare VideoAds against existing benchmarks in terms of video complexity. Through extensive experiments, we find that Qwen2.5-VL-72B, an opensource MLLM, achieves 73.35\% accuracy on VideoAds, outperforming GPT-4o (66.82\%) and Gemini-1.5 Pro (69.66\%); the two proprietary models especially fall behind the opensource model in video summarization and reasoning, but perform the best in visual finding. Notably, human experts easily achieve a remarkable accuracy of 94.27\%. These results underscore the necessity of advancing MLLMs' temporal modeling capabilities and highlight VideoAds as a potentially pivotal benchmark for future research in understanding video that requires high FPS sampling. The dataset and evaluation code will be publicly available at https://videoadsbenchmark.netlify.app.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。