一种多智能体感知-行动联盟用于高效长视频推理
A Multi-Agent Perception-Action Alliance for Efficient Long Video Reasoning
AI总结：
本文提出A4VL多智能体感知-行动探索联盟，通过多轮感知-行动探索循环提升长视频推理效率，采用事件驱动划分和线索引导块对齐技术，实现高质量推理并降低推理延迟。
摘要：
本文提出了一种名为A4VL的多智能体感知-行动探索联盟，用于高效长视频推理。A4VL在多轮感知-行动探索循环中运作，采用选定的视频语言模型（VLM）智能体。每轮中，智能体团队通过感知探索和行动探索执行视频问题-答案（VideoQA）任务。在感知探索阶段，每个智能体学习从少量采样的帧中提取查询特定的感知线索，并通过线索引导对齐找到最相关的视频块。在行动探索阶段，A4VL通过三个步骤进行视频推理：（1）每个智能体生成初始答案；（2）所有智能体通过交叉评审和相关性排名协作评分；（3）根据是否达成满意共识，决定是通过修剪（如过滤最低性能智能体）和重新编排（如基于新线索和匹配块的感知-行动探索）启动新轮次，或通过生成最终答案结束。通过多轮感知-行动探索整合多智能体联盟，结合事件驱动划分和线索引导块对齐，使A4VL能够有效扩展到现实世界的长视频，同时保持高质量视频推理。在五个流行的VideoQA基准测试中，A4VL优于18种现有的代表性VLM和11种针对长视频推理优化的方法，同时实现了显著更低的推理延迟。我们的代码已发布在https://github.com/git-disl/A4VL。
英文摘要：
This paper presents a multi-agent perception-action exploration alliance, dubbed A4VL, for efficient long-video reasoning. A4VL operates in a multi-round perception-action exploration loop with a selection of VLM agents. In each round, the team of agents performs video question-answer (VideoQA) via perception exploration followed by action exploration. During perception exploration, each agent learns to extract query-specific perception clue(s) from a few sampled frames and performs clue-based alignment to find the video block(s) that are most relevant to the query-specific event. During action exploration, A4VL performs video reasoning in three steps: (1) each agent produces its initial answer with rational, (2) all agents collaboratively scores one another through cross-reviews and relevance ranking, and (3) based on whether a satisfactory consensus is reached, the decision is made either to start a new round of perception-action deliberation by pruning (e.g., filtering out the lowest performing agent) and re-staging (e.g., new-clue and matching block based perception-action exploration), or to conclude by producing its final answer. The integration of the multi-agent alliance through multi-round perception-action exploration, coupled with event-driven partitioning and cue-guided block alignment, enables A4VL to effectively scale to real world long videos while preserving high quality video reasoning. Evaluation Results on five popular VideoQA benchmarks show that A4VL outperforms 18 existing representative VLMs and 11 recent methods optimized for long-video reasoning, while achieving significantly lower inference latency. Our code is released at https://github.com/git-disl/A4VL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
