Kangaroo:一种支持长上下文视频输入的强大视频-语言模型
Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Input
AI总结:
本文提出Kangaroo视频多模态模型,通过构建高质量数据集和逐步提高分辨率与帧数的课程训练流程,有效解决了长视频输入难题,以8B参数在长视频理解基准上超越10B以上大模型。
摘要:
在将大型语言模型(LLMs)扩展为大型多模态模型方面已取得快速进展。然而,将LLMs的输入模态扩展到视频数据仍是一项具有挑战性的工作,尤其是对于长视频。由于难以获取大规模高质量视频数据以及视觉特征的过度压缩,当前方法在有效处理长视频方面表现出局限性。在本文中,我们介绍了Kangaroo,一种旨在解决这些挑战的强大视频LMM。面对训练数据不足的问题,我们开发了一个数据筛选系统,以构建一个带有高质量标注的大规模数据集,用于视觉-语言预训练和指令微调。此外,我们设计了一个课程训练流程,通过逐步提高分辨率和输入帧数来适应长视频。评估结果表明,在拥有8B参数的情况下,Kangaroo在各种视频理解基准上取得了最先进的性能,同时在其他基准上也表现出具有竞争力的结果。特别是在专为长视频设计的基准上,Kangaroo超越了一些参数超过10B的更大模型和专有模型。
英文摘要:
Rapid advancements have been made in extending Large Language Models (LLMs) to Large Multi-modal Models (LMMs). However, extending input modality of LLMs to video data remains a challenging endeavor, especially for long videos. Due to insufficient access to large-scale high-quality video data and the excessive compression of visual features, current methods exhibit limitations in effectively processing long videos. In this paper, we introduce Kangaroo, a powerful Video LMM aimed at addressing these challenges. Confronted with issue of inadequate training data, we develop a data curation system to build a large-scale dataset with high-quality annotations for vision-language pre-training and instruction tuning. In addition, we design a curriculum training pipeline with gradually increasing resolution and number of input frames to accommodate long videos. Evaluation results demonstrate that, with 8B parameters, Kangaroo achieves state-of-the-art performance across a variety of video understanding benchmarks while exhibiting competitive results on others. Particularly, on benchmarks specialized for long videos, Kangaroo excels some larger models with over 10B parameters and proprietary models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。