基于定位的视频字幕生成
Grounded Video Caption Generation
AI总结：
本文提出了grounded video caption generation新任务，构建了GROC测试集和HowToGround训练集，并开发了VideoGround模型，通过统一视频字幕生成与目标定位，在时间一致的边界框基础上实现了该任务的最新技术水平。
摘要：
我们提出了一项新任务、数据集和模型，用于grounded video caption generation（基于定位的视频字幕生成）。该任务统一了视频中的字幕生成与目标定位，其中字幕中的目标通过时间一致的边界框在视频中被定位。我们引入了以下贡献。首先，我们提出了该任务的任务定义和一个人工标注的测试数据集，称为GROunded Video Caption Generation (GROC)。其次，我们引入了一种大规模自动标注方法，利用现有的grounded still image captioning模型，结合LLM将帧级字幕总结为视频中时间一致的字幕。此外，我们提示LLM通过语言进行跟踪——将帧级字幕中的名词短语分类为视频级生成字幕中的名词短语。我们将此方法应用于HowTo100M数据集中的视频，生成了一个新的大规模训练数据集，称为HowToGround，其中包含自动标注的字幕和时空一致的边界框，并附有连贯的自然语言标签。第三，我们引入了一种新的grounded video caption generation模型，称为VideoGround，并在新的自动标注的HowToGround数据集上训练该模型。最后，我们的VideoGround模型的结果为grounded video caption generation这一新任务设定了最新技术水平。我们进行了广泛的消融实验，并展示了模型关键技术贡献的重要性。
英文摘要：
We propose a new task, dataset and model for grounded video caption generation. This task unifies captioning and object grounding in video, where the objects in the caption are grounded in the video via temporally consistent bounding boxes. We introduce the following contributions. First, we present a task definition and a manually annotated test dataset for this task, referred to as GROunded Video Caption Generation (GROC). Second, we introduce a large-scale automatic annotation method leveraging an existing model for grounded still image captioning together with an LLM for summarising frame-level captions into temporally consistent captions in video. Furthermore, we prompt the LLM to track by language -- classifying noun phrases from the frame-level captions into noun phrases of the video-level generated caption. We apply this approach to videos from the HowTo100M dataset, which results in a new large-scale training dataset, called HowToGround, with automatically annotated captions and spatio-temporally consistent bounding boxes with coherent natural language labels. Third, we introduce a new grounded video caption generation model, called VideoGround, and train the model on the new automatically annotated HowToGround dataset. Finally, results of our VideoGround model set the state of the art for the new task of grounded video caption generation. We perform extensive ablations and demonstrate the importance of key technical contributions of our model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
