arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.06182cs.CV
收藏

通过细粒度视频故事生成实现长视频理解

Towards Long Video Understanding via Fine-detailed Video Story Generation

South China University of Technology(华南理工大学);Peng Cheng Laboratory(鹏城实验室);Shenzhen MSU-BIT University(深圳北理莫斯科大学);Shenzhen University(深圳大学)

Zeng You, Zhiquan Wen, Yaofo Chen, Xin Li, Runhao Zeng, Yaowei Wang, Mingkui Tan

AI总结:

针对长视频理解中长上下文建模难、冗余干扰大的问题,提出FDVS方法,通过自底向上解读机制和语义冗余消减机制生成层级化文本表示,无需微调即可适配多任务,在8个数据集上验证了有效性与通用性。

摘要:

长视频理解已成为计算机视觉领域的一项关键任务,推动了从监控到内容检索等众多应用的发展。现有视频理解方法在处理长视频理解时面临两大挑战:复杂的长上下文关系建模,以及冗余信息带来的干扰。为解决这些挑战,我们提出细粒度视频故事生成方法(Fine-Detailed Video Story generation,简称FDVS),可将长视频转化为详细的文本表示。具体而言,为实现对长时序内容的细粒度建模,我们提出自底向上的视频解读机制,从片段到视频逐步解读视频内容。为避免视频中冗余信息的干扰,我们引入语义冗余消减机制,在视觉和文本两个层面去除冗余。我们的方法将长视频转化为包含视频多粒度信息的层级化文本表示。依托这些表示,FDVS无需任何微调即可应用于多种任务。我们在涵盖三类任务的八个数据集上对所提方法进行了评估,实验结果验证了该方法的有效性与通用性。

英文摘要:

Long video understanding has become a critical task in computer vision, driving advancements across numerous applications from surveillance to content retrieval. Existing video understanding methods suffer from two challenges when dealing with long video understanding: intricate long-context relationship modeling and interference from redundancy. To tackle these challenges, we introduce Fine-Detailed Video Story generation (FDVS), which interprets long videos into detailed textual representations. Specifically, to achieve fine-grained modeling of long-temporal content, we propose a Bottom-up Video Interpretation Mechanism that progressively interprets video content from clips to video. To avoid interference from redundant information in videos, we introduce a Semantic Redundancy Reduction mechanism that removes redundancy at both the visual and textual levels. Our method transforms long videos into hierarchical textual representations that contain multi-granularity information of the video. With these representations, FDVS is applicable to various tasks without any fine-tuning. We evaluate the proposed method across eight datasets spanning three tasks. The performance demonstrates the effectiveness and versatility of our method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑