SOVC：面向主体的视频字幕生成
Towards Subject-Oriented Video Captioning via User-Specified Targets
AI总结：
提出面向主体的视频字幕生成任务 SOVC，构建基于 MSVD 与 MSRVTT 的数据集，并设计 SOVCNet 通过主体相关采样及硬、软提示编码提升目标描述效果。
摘要：
根据用户需求描述视频内容是一个长期追求的目标。尽管现有视频字幕生成方法已经取得显著进展，但生成的字幕可能并未聚焦于用户特别感兴趣的实体。为解决这一问题，我们提出了一项新的视频字幕生成任务——面向主体的视频字幕生成（Subject-Oriented Video Captioning，SOVC），旨在让用户通过边界框指定描述目标。为支持该任务，我们基于两个广泛使用的视频字幕生成数据集 MSVD 和 MSRVTT，通过为每条字幕标注每个视频中的主体，构建了两个面向主体的视频字幕生成数据集。这些数据集为描述用户感兴趣的目标铺平了道路。为完成该任务，我们引入了一种为其量身定制的方法，名为 SOVCNet。该方法包含两个关键组件：一是面向主体的采样模块，用于采样与主体相关的帧，以尽量减少无关信息；二是面向主体的编码模块，将主体区域作为硬提示，并整合可学习的软提示，从而增强模型对主体活动的关注，并促进其适应下游生成任务。大量实验结果证明了我们的方法在这一新任务上的有效性。
英文摘要：
Describing video content according to users' needs is a long-held goal. Although existing video captioning methods have made significant progress, the generated captions may not focus on the entity that users are particularly interested in. To address this problem, we propose a new video captioning task, Subject-Oriented Video Captioning (SOVC), which aims to allow users to specify the describing target via a bounding box. To support this task, we construct two subject-oriented video captioning datasets based on two widely used video captioning datasets: MSVD and MSRVTT, by annotating subjects in each video for each caption. These datasets pave the way for describing users' interested targets. To tackle this task, we introduce a method tailored to this task, named SOVCNet. It consists of two key components: a subject-oriented sampling module that samples frames related to the subject to minimize irrelevant information; and a subject-oriented encoding module that utilizes the subject areas as hard prompts and integrates learnable soft prompts, enhancing the model's focus on the subject's activities and facilitating adaptation to the downstream generation task. Extensive experimental results demonstrate the effectiveness of our method on this new task.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
