arXiv 2503.107812025-09-09cs.CV

大规模预训练用于接地视频字幕生成

Large-scale Pre-training for Grounded Video Caption Generation

Czech Institute of Informatics, Robotics and Cybernetics at the Czech Technical University in Prague（捷克信息技术、机器人与自动化研究所（捷克技术大学））；Inria, École normale supérieure, CNRS, PSL Research University（法国国家信息与自动化研究所、法国高等师范学校、法国国家科学研究中心、巴黎-萨克勒大学）

Evangelos Kazakos, Cordelia Schmid, Josef Sivic

备注：: Accepted at ICCV 2025. Erratum: An earlier version reported ablations (Table 6 & Fig. 6) with pre-training on a 50k subset of HowToGround1M + fine-tuning on iGround. In the ICCV camera-ready, Table 6 already used the full dataset, but Fig. 6 and a sentence in the text were mistakenly left on 50k. All now use the full HowToGround1M

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出大规模预训练方法，构建HowToGround1M数据集和GROVE模型，实现视频字幕生成与对象接地，在多个数据集上达到最先进性能。

摘要：

我们提出了一种新颖的视频字幕生成与对象接地方法，其中字幕中的对象通过时间上密集的边界框在视频中进行接地。我们做出以下贡献。首先，我们提出了一种大规模自动标注方法，将带有边界框接地的帧级字幕聚合为时间上密集且一致的标注。我们将此方法应用于HowTo100M数据集，构建了一个大规模预训练数据集，命名为HowToGround1M。我们还引入了一个接地视频字幕生成模型，称为GROVE，并在HowToGround1M上对该模型进行预训练。其次，我们引入了iGround——一个包含3513个视频的数据集，具有人工标注的字幕和密集的时空接地边界框。这使我们能够衡量这一具有挑战性问题的进展，并在此小规模但高质量的数据上微调我们的模型。第三，我们证明了我们的方法在提出的iGround数据集以及VidSTG、ActivityNet-Entities、GroundingYouTube和YouCook-Interactions数据集上达到了最先进的结果。我们的消融实验证明了在自动标注的HowToGround1M数据集上进行预训练，然后在人工标注的iGround数据集上进行微调的重要性，并验证了我们模型的关键技术贡献。数据集和代码可在https://ekazakos.github.io/grounded_video_caption_generation/获取。

英文摘要：

We propose a novel approach for captioning and object grounding in video, where the objects in the caption are grounded in the video via temporally dense bounding boxes. We introduce the following contributions. First, we present a large-scale automatic annotation method that aggregates frame-level captions grounded with bounding boxes into temporally dense and consistent annotations. We apply this approach on the HowTo100M dataset to construct a large-scale pre-training dataset, named HowToGround1M. We also introduce a Grounded Video Caption Generation model, dubbed GROVE, and pre-train the model on HowToGround1M. Second, we introduce iGround--a dataset of 3513 videos with manually annotated captions and dense spatio-temporally grounded bounding boxes. This allows us to measure progress on this challenging problem, as well as to fine-tune our model on this small-scale but high-quality data. Third, we demonstrate that our approach achieves state-of-the-art results on the proposed iGround dataset, as well as on the VidSTG, ActivityNet-Entities, GroundingYouTube, and YouCook-Interactions datasets. Our ablations demonstrate the importance of pre-training on our automatically annotated HowToGround1M dataset followed by fine-tuning on the manually annotated iGround dataset and validate the key technical contributions of our model. The dataset and code are available at https://ekazakos.github.io/grounded_video_caption_generation/.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。