流式密集视频字幕生成
Streaming Dense Video Captioning
AI总结：
本文提出流式密集视频字幕生成模型，通过固定大小聚类记忆模块和流式解码算法，实现处理任意长视频并提前输出预测，在ActivityNet、YouCook2和ViTT三个基准上显著超越现有最先进方法。
摘要：
理想的密集视频字幕生成模型——预测视频中时间上定位的字幕——应能处理长输入视频，预测丰富、详细的文本描述，并能在处理完整视频之前产生输出。然而，当前最先进的模型处理固定数量的下采样帧，并在看完整个视频后做出单一完整预测。我们提出了一种流式密集视频字幕生成模型，包含两个新颖组件：第一，我们提出了一种新的记忆模块，基于对传入token进行聚类，由于记忆大小固定，可以处理任意长度的视频。第二，我们开发了一种流式解码算法，使我们的模型能够在整个视频处理完毕之前做出预测。我们的模型实现了这种流式能力，并在三个密集视频字幕生成基准上显著提升了最先进水平：ActivityNet、YouCook2和ViTT。我们的代码已发布在 https://github.com/google-research/scenic。
英文摘要：
An ideal model for dense video captioning -- predicting captions localized temporally in a video -- should be able to handle long input videos, predict rich, detailed textual descriptions, and be able to produce outputs before processing the entire video. Current state-of-the-art models, however, process a fixed number of downsampled frames, and make a single full prediction after seeing the whole video. We propose a streaming dense video captioning model that consists of two novel components: First, we propose a new memory module, based on clustering incoming tokens, which can handle arbitrarily long videos as the memory is of a fixed size. Second, we develop a streaming decoding algorithm that enables our model to make predictions before the entire video has been processed. Our model achieves this streaming ability, and significantly improves the state-of-the-art on three dense video captioning benchmarks: ActivityNet, YouCook2 and ViTT. Our code is released at https://github.com/google-research/scenic.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
