arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.16620cs.CVcs.CL
收藏

OmAgent:基于任务分治的复杂视频理解多模态智能体框架

OmAgent: A Multi-modal Agent Framework for Complex Video Understanding with Task Divide-and-Conquer

Om AI Research(Om AI研究院);Binjiang Institute of Zhejiang University(浙江大学滨江研究院)

Lu Zhang, Tiancheng Zhao, Heting Ying, Yibo Ma, Kyusong Lee

AI总结:

针对长视频处理信息丢失问题,提出多模态智能体框架OmAgent,通过高效存储检索视频帧与分治循环动态调用工具,显著减少信息丢失并提升复杂视频理解准确性。

摘要:

大型语言模型(LLMs)的最新进展扩展了其在多模态上下文中的能力,包括全面的视频理解。然而,处理诸如24小时CCTV录像或完整长片电影等冗长视频,由于庞大的数据和处理需求而面临重大挑战。传统方法(如提取关键帧或将帧转换为文本)通常会导致大量信息丢失。为了解决这些缺陷,我们开发了OmAgent,它能高效地存储和检索与特定查询相关的视频帧,从而保留视频的详细内容。此外,它具备一个分治循环(Divide-and-Conquer Loop),能够进行自主推理,动态调用API和工具以增强查询处理和准确性。这种方法确保了稳健的视频理解,显著减少了信息丢失。实验结果证实了OmAgent在处理各类视频和复杂任务方面的有效性。此外,我们赋予了它更高的自主性和稳健的工具调用系统,使其能够完成更为复杂的任务。

英文摘要:

Recent advancements in Large Language Models (LLMs) have expanded their capabilities to multimodal contexts, including comprehensive video understanding. However, processing extensive videos such as 24-hour CCTV footage or full-length films presents significant challenges due to the vast data and processing demands. Traditional methods, like extracting key frames or converting frames to text, often result in substantial information loss. To address these shortcomings, we develop OmAgent, efficiently stores and retrieves relevant video frames for specific queries, preserving the detailed content of videos. Additionally, it features an Divide-and-Conquer Loop capable of autonomous reasoning, dynamically invoking APIs and tools to enhance query processing and accuracy. This approach ensures robust video understanding, significantly reducing information loss. Experimental results affirm OmAgent's efficacy in handling various types of videos and complex tasks. Moreover, we have endowed it with greater autonomy and a robust tool-calling system, enabling it to accomplish even more intricate tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑