arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2312.17432cs.CVcs.CL
收藏

利用大语言模型进行视频理解:综述

Video Understanding with Large Language Models: A Survey

University of Rochester(罗切斯特大学);The University of Hong Kong(香港大学);Southern University of Science and Technology(南方科技大学)
Yolo Y. Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, Ali Vosoughi, Chao Huang, Zeliang Zhang, … 展开作者

Yolo Y. Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, Ali Vosoughi, Chao Huang, Zeliang Zhang, Pinxin Liu, Mingqian Feng, Feng Zheng, Jianguo Zhang, Ping Luo, Jiebo Luo, Chenliang Xu

AI总结:

本文综述了利用大语言模型进行视频理解的最新进展,探讨了Vid-LLMs的分类、功能及应用场景,并指出了未来研究方向。

摘要:

随着在线视频平台的快速增长和视频内容的不断增长,对高效视频理解工具的需求显著增加。鉴于大语言模型(LLMs)在语言和多模态任务中的出色能力,本文综述了近年来利用LLMs(Vid-LLMs)进行视频理解的最新进展。Vid-LLMs展现出惊人的能力,特别是其结合常识知识的开放式多粒度(一般、时间性和时空)推理能力,表明了未来视频理解的一个有前景的路径。我们探讨了Vid-LLMs的独特特性和能力,将其方法分为三种主要类型:视频分析器xLLM、视频嵌入器xLLM以及(分析器+嵌入器)xLLM。此外,我们根据Vid-LLMs中LLM的功能,将其分为五种子类型:LLM作为摘要器、LLM作为管理者、LLM作为文本解码器、LLM作为回归器和LLM作为隐藏层。此外,本文还全面研究了Vid-LLMs的任务、数据集、基准和评估方法。同时,它探索了Vid-LLMs在各种领域中的广泛应用,突显了其在现实世界视频理解挑战中的卓越可扩展性和多功能性。最后,它总结了现有Vid-LLMs的局限性,并概述了未来研究的方向。如需更多信息,请访问仓库:https://github.com/yunlong10/Awesome-LLMs-for-Video-Understanding。

英文摘要:

With the burgeoning growth of online video platforms and the escalating volume of video content, the demand for proficient video understanding tools has intensified markedly. Given the remarkable capabilities of large language models (LLMs) in language and multimodal tasks, this survey provides a detailed overview of recent advancements in video understanding that harness the power of LLMs (Vid-LLMs). The emergent capabilities of Vid-LLMs are surprisingly advanced, particularly their ability for open-ended multi-granularity (general, temporal, and spatiotemporal) reasoning combined with commonsense knowledge, suggesting a promising path for future video understanding. We examine the unique characteristics and capabilities of Vid-LLMs, categorizing the approaches into three main types: Video Analyzer x LLM, Video Embedder x LLM, and (Analyzer + Embedder) x LLM. Furthermore, we identify five sub-types based on the functions of LLMs in Vid-LLMs: LLM as Summarizer, LLM as Manager, LLM as Text Decoder, LLM as Regressor, and LLM as Hidden Layer. Furthermore, this survey presents a comprehensive study of the tasks, datasets, benchmarks, and evaluation methodologies for Vid-LLMs. Additionally, it explores the expansive applications of Vid-LLMs across various domains, highlighting their remarkable scalability and versatility in real-world video understanding challenges. Finally, it summarizes the limitations of existing Vid-LLMs and outlines directions for future research. For more information, readers are recommended to visit the repository at https://github.com/yunlong10/Awesome-LLMs-for-Video-Understanding.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑