arXiv 2605.286042026-05-27cs.CVcs.AI

挖掘多模态时空线索用于视频重要人物识别

Mining Multi-Modality Spatio-Temporal Cues for Video Important Person Identification

School of Computer Science and Technology, Wuhan University of Science and Technology（武汉科技大学计算机科学与技术学院）；Hubei Province Key Laboratory of Intelligent Information Processing and Real-time Industrial System, Wuhan University of Science and Technology（湖北省智能信息处理与实时工业系统重点实验室）；School of Computer Science, National Engineering Research Center for Multimedia Software, Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University（计算机科学学院，国家多媒体软件工程技术研究中心，湖北省多媒体与网络通信工程重点实验室，武汉大学）；College of Computing and Data Science, Nanyang Technological University（计算与数据科学学院，南洋理工大学）

Xiao Wang, Minglei Yang, Bin Yang, Wenke Huang, Zheng Wang, Xin Xu, Mang Ye

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对视频中人物重要性随时间变化的问题，提出VIP-Net框架，通过多模态时空线索融合与时间重要性矫正，在Temporal-VIP数据集上达到67.3%准确率。

摘要：

识别视频场景中的关键人物对于自动视频编辑和智能监控等应用至关重要。当前方法主要关注静态图像和即时视觉线索，忽略了视频中丰富的时空信息。这导致了时间重要性转移（TIS）现象，即早期帧中被认为重要的人物在考虑整个时间上下文后可能被降级。为了解决这一问题，我们引入了视频重要人物（VIP）识别任务，旨在自动识别视频中最具影响力的人物，同时提供文本理由。我们提出了Temporal-VIP，一个大规模的理由标注数据集，包含11个类别的9,249个视频片段，并附有对齐的重要性理由。为了缓解TIS，我们开发了VIP-Net框架，包括用于提取多模态时空线索的社会线索编码器（SCE）、用于层次化线索融合和跨模态对齐的时间重要性矫正器（TIR），以及用于人物排序的VIP推理。实验结果表明，VIP-Net达到了67.3%的准确率，显著优于最先进的模型（37.5%-53.9%），并通过特征引导的LLM优化，平均理由相似度达到0.63。数据集和代码可在https://huggingface.co/datasets/yml2002/Temporal-VIP获取。

英文摘要：

Identifying key individuals in video scenes is essential for applications such as automated video editing and intelligent surveillance. Current methods primarily focus on static images and immediate visual cues, overlooking the rich spatio-temporal information in videos. This leads to the phenomenon of Temporal Importance Shift (TIS), wherein individuals deemed significant in early frames may be demoted as the entire temporal context is considered. To address this, we introduce the Video Important Person (VIP) identification task, aimed at automatically identifying the most influential individuals in videos while providing textual rationales. We present Temporal-VIP, a large-scale rationale-annotated dataset consisting of 9,249 video segments across 11 categories with aligned importance rationales. To mitigate TIS, we develop the VIP-Net framework, which includes a Social Cue Encoder (SCE) for extracting multi-modal spatio-temporal cues, a Temporal Importance Rectifier (TIR) for hierarchical cue fusion and cross-modal alignment, and VIP Inference for ranking individuals. Experimental results show that VIP-Net achieves 67.3% accuracy, significantly outperforming state-of-the-art models (37.5%-53.9%) and yielding a mean rationale similarity of 0.63 to ground truth through feature-guided LLM refinement. The dataset and code are available at https://huggingface.co/datasets/yml2002/Temporal-VIP.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。