arXiv 2411.130932025-11-23cs.CVcs.AI

Video-RAG: 基于视觉对齐的检索增强长视频理解

Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension

Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（中国教育部多媒体可信感知与高效计算重点实验室，厦门大学）；Nanjing University（南京大学）；University of Rochester（罗切斯特大学）

Yongdong Luo, Xiawu Zheng, Guilin Li, Shukang Yin, Haojia Lin, Chaoyou Fu, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, Rongrong Ji

备注：: Accepted at NeurIPS 2025. Camera-ready version

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

Video-RAG通过视觉对齐的辅助文本提升长视频理解性能，无需训练且兼容性强，显著优于专有模型。

摘要：

现有的大型视频-语言模型（LVLMs）在正确理解长视频方面存在困难，主要是由于上下文有限。为了解决这个问题，微调长上下文LVLMs和采用基于GPT的代理已成为有前途的解决方案。然而，微调LVLMs需要大量高质量数据和大量GPU资源，而基于GPT的代理则依赖于专有模型（例如GPT-4o）。在本文中，我们提出了Video Retrieval-Augmented Generation（Video-RAG），一种无需训练且成本效益高的流程，它利用视觉对齐的辅助文本来帮助促进跨模态对齐，同时提供超出视觉内容的信息。具体来说，我们利用开源外部工具从纯视频数据中提取视觉对齐的信息（例如音频、光学字符和物体检测），并将提取的信息整合到现有的LVLM中作为辅助文本，同时与视频帧和查询一起以即插即用的方式使用。我们的Video-RAG提供了几个关键优势：（i）轻量级，由于单轮检索，计算开销低；（ii）易于实现且兼容任何LVLM；（iii）在长视频理解基准测试中，包括Video-MME、MLVU和LongVideoBench，都实现了显著且一致的性能提升。值得注意的是，当与72B模型一起使用时，我们的模型在专有模型如Gemini-1.5-Pro和GPT-4o上表现出优越的性能。

英文摘要：

Existing large video-language models (LVLMs) struggle to comprehend long videos correctly due to limited context. To address this problem, fine-tuning long-context LVLMs and employing GPT-based agents have emerged as promising solutions. However, fine-tuning LVLMs would require extensive high-quality data and substantial GPU resources, while GPT-based agents would rely on proprietary models (e.g., GPT-4o). In this paper, we propose Video Retrieval-Augmented Generation (Video-RAG), a training-free and cost-effective pipeline that employs visually-aligned auxiliary texts to help facilitate cross-modality alignment while providing additional information beyond the visual content. Specifically, we leverage open-source external tools to extract visually-aligned information from pure video data (e.g., audio, optical character, and object detection), and incorporate the extracted information into an existing LVLM as auxiliary texts, alongside video frames and queries, in a plug-and-play manner. Our Video-RAG offers several key advantages: (i) lightweight with low computing overhead due to single-turn retrieval; (ii) easy implementation and compatibility with any LVLM; and (iii) significant, consistent performance gains across long video understanding benchmarks, including Video-MME, MLVU, and LongVideoBench. Notably, our model demonstrates superior performance over proprietary models like Gemini-1.5-Pro and GPT-4o when utilized with a 72B model.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。