arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2504.04471cs.CV
收藏

VideoAgent2:通过不确定性感知的思维链增强基于大语言模型的智能体系统以用于长视频理解

VideoAgent2: Enhancing the LLM-Based Agent System for Long-Form Video Understanding by Uncertainty-Aware CoT

University College London(伦敦大学学院);City University of Hong Kong(香港城市大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室)

Zhuo Zhi, Qiangqiang Wu, Minghe shen, Wenbo Li, Yinchuan Li, Kun Shao, Kaiwen Zhou

AI总结:

本文提出VideoAgent2系统,通过引入不确定性感知的思维链(CoT)和计划调整模式,增强LLM在长视频理解中的推理能力并抑制外部工具噪声,在多个基准上平均超越VideoAgent 13.1%。

摘要:

长视频理解已成为计算机视觉中日益重要但极具挑战性的任务。基于智能体的方法在处理长视频方面日益受到关注,因为它们能够处理扩展序列并整合多种工具以捕获细粒度信息。然而,现有方法仍面临若干挑战:(1)它们通常仅依赖大语言模型(LLMs)的推理能力,缺乏专门机制来增强长视频场景下的推理;(2)它们仍然容易受到外部工具错误或噪声的影响。为解决这些问题,我们提出了一种专为长视频分析定制的思维链(CoT)过程。我们提出的具有计划调整模式的CoT使LLM能够逐步规划并调整其信息收集策略。我们进一步引入对LLM和外部工具的启发式不确定性估计来指导CoT过程。这使得LLM能够评估新收集信息的可靠性,改进其收集策略,并在综合最终答案时做出更稳健的决策。实证实验表明,我们的不确定性感知CoT有效缓解了来自外部工具的噪声,从而产生更可靠的输出。我们在名为VideoAgent2的系统中实现了该方法,该系统还包括通用上下文获取和专用工具设计等附加模块。在三个专门的长视频基准(及其子集)上的评估表明,VideoAgent2比先前最先进的基于智能体的方法VideoAgent平均高出13.1%,并在所有零样本方法中取得了领先性能。

英文摘要:

Long video understanding has emerged as an increasingly important yet challenging task in computer vision. Agent-based approaches are gaining popularity for processing long videos, as they can handle extended sequences and integrate various tools to capture fine-grained information. However, existing methods still face several challenges: (1) they often rely solely on the reasoning ability of large language models (LLMs) without dedicated mechanisms to enhance reasoning in long video scenarios; and (2) they remain vulnerable to errors or noise from external tools. To address these issues, we propose a specialized chain-of-thought (CoT) process tailored for long video analysis. Our proposed CoT with plan-adjust mode enables the LLM to incrementally plan and adapt its information-gathering strategy. We further incorporate heuristic uncertainty estimation of both the LLM and external tools to guide the CoT process. This allows the LLM to assess the reliability of newly collected information, refine its collection strategy, and make more robust decisions when synthesizing final answers. Empirical experiments show that our uncertainty-aware CoT effectively mitigates noise from external tools, leading to more reliable outputs. We implement our approach in a system called VideoAgent2, which also includes additional modules such as general context acquisition and specialized tool design. Evaluation on three dedicated long video benchmarks (and their subsets) demonstrates that VideoAgent2 outperforms the previous state-of-the-art agent-based method, VideoAgent, by an average of 13.1% and achieves leading performance among all zero-shot approaches

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑