arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.03104cs.CVcs.CLcs.MM
收藏

KeyVideoLLM:面向大规模视频关键帧选择

KeyVideoLLM: Towards Large-scale Video Keyframe Selection

Peking University(北京大学);The Open University of China(中国开放大学);Huawei Cloud BU(华为云业务部);Shanghai AI Laboratory(上海人工智能实验室);University of Chinese Academy of Sciences(中国科学院大学)

Hao Liang, Jiapeng Li, Tianyi Bai, Xijie Huang, Linzhuang Sun, Zhengren Wang, Conghui He, Bin Cui, Chong Chen, Wentao Zhang

AI总结:

针对视频大语言模型数据管理的效率、鲁棒性和有效性挑战,提出基于文本-视频帧相似度的关键帧选择方法KeyVideoLLM,实现高压缩率与快处理速度,且提升视频问答任务性能并达SoTA。

摘要:

近年来,随着网络视频的兴起,管理和理解大规模视频数据集变得日益重要。视频大语言模型(VideoLLMs)凭借其强大的视频理解能力在近年涌现。然而,VideoLLMs的训练和推理过程需要海量数据,给数据管理带来了效率、鲁棒性和有效性方面的重大挑战。本研究提出KeyVideoLLM,这是一种基于文本-视频帧相似度的关键帧选择方法,旨在高效、鲁棒且有效地管理VideoLLM数据。具体而言,KeyVideoLLM实现了高达60.9倍的显著数据压缩率,大幅降低了磁盘空间需求,证明了其高效性。此外,它在所有视频格式和规模下都保持100%的选择成功率,相比现有关键帧选择方法处理速度提升最高达200倍,且无需超参数调优。除了出色的效率与鲁棒性,KeyVideoLLM还进一步提升了模型在视频问答任务训练和推理阶段的性能,值得注意的是,它在多个数据集上均取得了当前最优(SoTA)的实验结果。

英文摘要:

Recently, with the rise of web videos, managing and understanding large-scale video datasets has become increasingly important. Video Large Language Models (VideoLLMs) have emerged in recent years due to their strong video understanding capabilities. However, training and inference processes for VideoLLMs demand vast amounts of data, presenting significant challenges to data management, particularly regarding efficiency, robustness, and effectiveness. In this work, we present KeyVideoLLM, a text-video frame similarity-based keyframe selection method designed to manage VideoLLM data efficiently, robustly, and effectively. Specifically, KeyVideoLLM achieves a remarkable data compression rate of up to 60.9 times, substantially lowering disk space requirements, which proves its high efficiency. Additionally, it maintains a 100% selection success rate across all video formats and scales, enhances processing speed by up to 200 times compared to existing keyframe selection methods, and does not require hyperparameter tuning. Beyond its outstanding efficiency and robustness, KeyVideoLLM further improves model performance in video question-answering tasks during both training and inference stages. Notably, it consistently achieved the state-of-the-art (SoTA) experimental results on diverse datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑