arXiv 2407.189082025-03-20cs.LGcs.CLcs.CV

Wolf：基于世界摘要框架的密集视频字幕生成

Wolf: Dense Video Captioning with a World Summarization Framework

NVIDIA（英伟达）；UC Berkeley（加州大学伯克利分校）；MIT（麻省理工学院）；UT Austin（德克萨斯大学奥斯汀分校）；University of Toronto（多伦多大学）；Stanford University（斯坦福大学）

Boyi Li, Ligeng Zhu, Ran Tian, Shuhan Tan, Yuxiao Chen, Yao Lu, Yin Cui, Sushant Veer, Max Ehrlich, Jonah Philion, Xinshuo Weng, Fuzhao Xue, Linxi Fan, Yuke Zhu… 展开作者

Boyi Li, Ligeng Zhu, Ran Tian, Shuhan Tan, Yuxiao Chen, Yao Lu, Yin Cui, Sushant Veer, Max Ehrlich, Jonah Philion, Xinshuo Weng, Fuzhao Xue, Linxi Fan, Yuke Zhu, Jan Kautz, Andrew Tao, Ming-Yu Liu, Sanja Fidler, Boris Ivanovic, Trevor Darrell, Jitendra Malik, Song Han, Marco Pavone

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究提出名为Wolf的世界摘要视频字幕生成框架，采用混合专家方法结合多类视觉语言模型优势，提出CapScore评估指标，构建多领域人工标注数据集，实验证明其性能优于现有最优方法与商业方案，还建立了基准与排行榜以推动领域发展。

摘要：

我们提出Wolf——一种用于精准视频字幕生成的WOrLd（世界）摘要框架。Wolf是一个自动化字幕生成框架，采用混合专家方法，充分利用视觉语言模型（Vision Language Models, VLMs）的互补优势。该框架同时运用图像模型和视频模型，捕捉不同层级的信息并进行高效摘要，可应用于增强视频理解、自动标注和字幕生成任务。为评估字幕质量，我们提出CapScore——一种基于大语言模型（LLM）的指标，用于衡量生成字幕与真实标注字幕的相似度和质量。我们进一步在自动驾驶、通用场景、机器人三个领域构建了四个人工标注数据集，以支持全面的性能对比。实验表明，与研究领域的现有最优方法（VILA1.5、CogAgent）以及商业解决方案（Gemini-Pro-1.5、GPT-4V）相比，Wolf实现了更优的字幕生成性能。例如，在具有挑战性的驾驶视频场景中，与GPT-4V相比，Wolf的CapScore在质量维度提升了55.6%，在相似度维度提升了77.4%。最后，我们建立了视频字幕生成基准并推出了排行榜，旨在推动视频理解、字幕生成和数据对齐领域的发展。项目网页：https://wolfv0.github.io/。

英文摘要：

We propose Wolf, a WOrLd summarization Framework for accurate video captioning. Wolf is an automated captioning framework that adopts a mixture-of-experts approach, leveraging complementary strengths of Vision Language Models (VLMs). By utilizing both image and video models, our framework captures different levels of information and summarizes them efficiently. Our approach can be applied to enhance video understanding, auto-labeling, and captioning. To evaluate caption quality, we introduce CapScore, an LLM-based metric to assess the similarity and quality of generated captions compared to the ground truth captions. We further build four human-annotated datasets in three domains: autonomous driving, general scenes, and robotics, to facilitate comprehensive comparisons. We show that Wolf achieves superior captioning performance compared to state-of-the-art approaches from the research community (VILA1.5, CogAgent) and commercial solutions (Gemini-Pro-1.5, GPT-4V). For instance, in comparison with GPT-4V, Wolf improves CapScore both quality-wise by 55.6% and similarity-wise by 77.4% on challenging driving videos. Finally, we establish a benchmark for video captioning and introduce a leaderboard, aiming to accelerate advancements in video understanding, captioning, and data alignment. Webpage: https://wolfv0.github.io/.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。