arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.04325cs.CV
收藏

ShareGPT4Video:通过更优字幕提升视频理解与生成

ShareGPT4Video: Improving Video Understanding and Generation with Better Captions

USTC(中国科学技术大学);CUHK(香港中文大学);PKU(北京大学);Shanghai AI Lab(上海人工智能实验室)

Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, Li Yuan, Yu Qiao, Dahua Lin, Feng Zhao, Jiaqi Wang

AI总结:

本文提出 ShareGPT4Video 系列,通过差分视频字幕策略生成密集精确字幕,构建 40K 高质量视频数据集并训练 ShareCaptioner-Video 和 ShareGPT4Video-8B,显著提升视频理解与生成性能。

摘要:

我们提出 ShareGPT4Video 系列,旨在通过密集且精确的字幕促进大型视频语言模型(LVLMs)的视频理解以及文本到视频模型(T2VMs)的视频生成。该系列包括:1)ShareGPT4Video,包含 40K 条由 GPT4V 标注的、覆盖不同长度和来源视频的密集字幕,通过精心设计的数据过滤和标注策略构建。2)ShareCaptioner-Video,一个高效且能力强的任意视频字幕生成模型,并用其为 4.8M 高质量美学视频标注字幕。3)ShareGPT4Video-8B,一个简单但出色的 LVLM,在三个前沿视频基准上达到了 SOTA 性能。为实现这一目标,在排除不可扩展且昂贵的人工标注者后,我们发现使用 GPT4V 以朴素的多帧或帧拼接输入策略为视频生成字幕会导致细节不足,有时还会产生时间混淆的结果。我们认为设计高质量视频字幕策略的挑战在于三个方面:1)帧间精确的时间变化理解。2)帧内详细的内容描述。3)面向任意长度视频的帧数可扩展性。为此,我们精心设计了一种差分视频字幕策略,该策略稳定、可扩展且高效,能够为任意分辨率、宽高比和长度的视频生成字幕。基于该策略,我们构建了 ShareGPT4Video,其中包含 40K 个覆盖广泛类别的高质量视频,所得字幕包含丰富的世界知识、物体属性、摄像机运动,以及关键的是对事件的详细且精确的时间描述。基于 ShareGPT4Video,我们进一步开发了 ShareCaptioner-Video,这是一个更优的字幕生成器,能够高效地为任意视频生成高质量字幕,并利用其为 4.8M 高质量美学视频标注字幕。在 ShareCaptioner-Video 生成的字幕上训练后,ShareGPT4Video-8B 在三个前沿视频基准上达到了 SOTA 性能。

英文摘要:

We present the ShareGPT4Video series, aiming to facilitate the video understanding of large video-language models (LVLMs) and the video generation of text-to-video models (T2VMs) via dense and precise captions. The series comprises: 1) ShareGPT4Video, 40K GPT4V annotated dense captions of videos with various lengths and sources, developed through carefully designed data filtering and annotating strategy. 2) ShareCaptioner-Video, an efficient and capable captioning model for arbitrary videos, with 4.8M high-quality aesthetic videos annotated by it. 3) ShareGPT4Video-8B, a simple yet superb LVLM that reached SOTA performance on three advancing video benchmarks. To achieve this, taking aside the non-scalable costly human annotators, we find using GPT4V to caption video with a naive multi-frame or frame-concatenation input strategy leads to less detailed and sometimes temporal-confused results. We argue the challenge of designing a high-quality video captioning strategy lies in three aspects: 1) Inter-frame precise temporal change understanding. 2) Intra-frame detailed content description. 3) Frame-number scalability for arbitrary-length videos. To this end, we meticulously designed a differential video captioning strategy, which is stable, scalable, and efficient for generating captions for videos with arbitrary resolution, aspect ratios, and length. Based on it, we construct ShareGPT4Video, which contains 40K high-quality videos spanning a wide range of categories, and the resulting captions encompass rich world knowledge, object attributes, camera movements, and crucially, detailed and precise temporal descriptions of events. Based on ShareGPT4Video, we further develop ShareCaptioner-Video, a superior captioner capable of efficiently generating high-quality captions for arbitrary videos...

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑