arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.10517cs.CVcs.AIcs.CLcs.IR
收藏

VideoAgent:以大型语言模型为智能体的长视频理解

VideoAgent: Long-form Video Understanding with Large Language Model as Agent

Stanford University(斯坦福大学)

Xiaohan Wang, Yuhui Zhang, Orr Zohar, Serena Yeung-Levy

AI总结:

针对长视频理解难题,提出以大型语言模型为中央智能体、视觉-语言模型为工具的VideoAgent系统,通过交互式推理与规划,在EgoSchema和NExT-QA上以少量帧取得领先的零样本准确率。

摘要:

长视频理解是计算机视觉领域中的一项重大挑战,要求模型能够对长时间的多模态序列进行推理。受人类认知过程对长视频理解的启发,我们强调交互式推理和规划,而非处理冗长视觉输入的能力。我们提出了一种新颖的基于智能体的系统VideoAgent,该系统将大型语言模型作为中心智能体,迭代地识别并整合关键信息以回答问题,并以视觉-语言基础模型作为工具来翻译和检索视觉信息。在具有挑战性的EgoSchema和NExT-QA基准上进行评估,VideoAgent分别实现了54.1%和71.3%的零样本准确率,且平均仅使用8.4和8.2帧。这些结果表明,我们的方法在有效性和效率上均优于当前最先进的方法,凸显了基于智能体的方法在推进长视频理解方面的潜力。

英文摘要:

Long-form video understanding represents a significant challenge within computer vision, demanding a model capable of reasoning over long multi-modal sequences. Motivated by the human cognitive process for long-form video understanding, we emphasize interactive reasoning and planning over the ability to process lengthy visual inputs. We introduce a novel agent-based system, VideoAgent, that employs a large language model as a central agent to iteratively identify and compile crucial information to answer a question, with vision-language foundation models serving as tools to translate and retrieve visual information. Evaluated on the challenging EgoSchema and NExT-QA benchmarks, VideoAgent achieves 54.1% and 71.3% zero-shot accuracy with only 8.4 and 8.2 frames used on average. These results demonstrate superior effectiveness and efficiency of our method over the current state-of-the-art methods, highlighting the potential of agent-based approaches in advancing long-form video understanding.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑