arXiv 2507.185312025-07-24cs.CV

IntentVCNet：弥合时空差距以实现面向意图的可控视频描述生成

IntentVCNet: Bridging Spatio-Temporal Gaps for Intention-Oriented Controllable Video Captioning

University of Science and Technology of China（中国科学技术大学）；Hefei Institutes of Physical Science,Chinese Academy of Sciences（合肥物理研究所，中国科学院）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（人工智能研究所，合肥综合性国家科学中心）；State Key Lab. for Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）；University of Chicago（芝加哥大学）；National University of Defense Technology（国防科技大学）；Harbin Institute of Technology（哈尔滨工业大学）

Tianheng Qiu, Jingchun Gao, Jingyu Li, Huiyi Leong, Xuan Huang, Xi Wang, Xiaocheng Zhang, Kele Xu, Lan Zhang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出IntentVCNet，通过提示组合策略与参数高效框适配器统一LVLMs的时空理解，实现细粒度面向意图的可控视频描述，并在多个开源模型上达到SOTA。

摘要：

面向意图的可控视频描述生成旨在根据定制的用户意图，为视频中的特定目标生成有针对性的描述。当前的大型视觉语言模型（Large Visual Language Models，LVLMs）已经具备强大的指令遵循和视觉理解能力。尽管LVLMs分别在空间和时间理解方面表现出熟练水平，但它们无法在时间序列中直接响应指令以执行细粒度的空间控制。这一显著的时空差距使视频中实现细粒度的面向意图控制变得复杂。为此，我们提出了一种新颖的IntentVCNet，它统一了LVLMs内在的时间和空间理解知识，从提示和模型两个角度弥合时空差距。具体而言，我们首先提出一种提示组合策略，旨在使LLM能够建模表征用户意图的提示与视频序列之间的隐式关系。随后，我们提出一种参数高效的框适配器，用于增强全局视觉上下文中的对象语义信息，从而使视觉token具备关于用户意图的先验信息。最终实验证明，两种策略的结合能够进一步增强LVLM对视频序列中空间细节的建模能力，并促进LVLMs准确生成受控的面向意图描述。我们提出的方法在多个开源LVLMs上取得了最先进结果，并在IntentVC挑战赛中获得亚军。我们的代码可在 https://github.com/thqiu0419/IntentVCNet 获取。

英文摘要：

Intent-oriented controlled video captioning aims to generate targeted descriptions for specific targets in a video based on customized user intent. Current Large Visual Language Models (LVLMs) have gained strong instruction following and visual comprehension capabilities. Although the LVLMs demonstrated proficiency in spatial and temporal understanding respectively, it was not able to perform fine-grained spatial control in time sequences in direct response to instructions. This substantial spatio-temporal gap complicates efforts to achieve fine-grained intention-oriented control in video. Towards this end, we propose a novel IntentVCNet that unifies the temporal and spatial understanding knowledge inherent in LVLMs to bridge the spatio-temporal gap from both prompting and model perspectives. Specifically, we first propose a prompt combination strategy designed to enable LLM to model the implicit relationship between prompts that characterize user intent and video sequences. We then propose a parameter efficient box adapter that augments the object semantic information in the global visual context so that the visual token has a priori information about the user intent. The final experiment proves that the combination of the two strategies can further enhance the LVLM's ability to model spatial details in video sequences, and facilitate the LVLMs to accurately generate controlled intent-oriented captions. Our proposed method achieved state-of-the-art results in several open source LVLMs and was the runner-up in the IntentVC challenge. Our code is available on https://github.com/thqiu0419/IntentVCNet.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。