视频之中有何物：面向在线密集视频描述的分因式自回归解码
Whats in a Video: Factorized Autoregressive Decoding for Online Dense Video Captioning
AI总结：
本文提出一种基于分因式自回归解码的在线密集视频描述方法，无需未来帧即可输出频繁且时间对齐的局部字幕，并通过训练推理优化支持长视频，性能优于现有方法且计算量降低20%。
摘要：
为视频自动生成能够准确描述其内容的密集字幕仍然是研究中具有挑战性的领域。当前大多数模型需要一次性处理整个视频。相反，我们提出一种高效的在线方法，在不访问未来帧的情况下，输出频繁、详细且时间对齐的字幕。我们的模型采用一种新颖的自回归分因式解码架构，对每个时间段视觉特征的序列进行建模，输出局部化描述，并高效利用来自先前视频片段的上下文。这使得模型能够根据视频实际的局部内容输出频繁、详细的字幕，从而更全面地描述视频，而不是模仿训练数据。其次，我们提出一种用于高效训练和推理的优化方法，使模型能够扩展到更长的视频。与离线和在线方法相比，我们的方法表现出优异的性能，并且计算量减少20%。所生成的标注更加全面和频繁，还可以进一步用于自动视频标记和大规模视频数据采集。
英文摘要：
Generating automatic dense captions for videos that accurately describe their contents remains a challenging area of research. Most current models require processing the entire video at once. Instead, we propose an efficient, online approach which outputs frequent, detailed and temporally aligned captions, without access to future frames. Our model uses a novel autoregressive factorized decoding architecture, which models the sequence of visual features for each time segment, outputting localized descriptions and efficiently leverages the context from the previous video segments. This allows the model to output frequent, detailed captions to more comprehensively describe the video, according to its actual local content, rather than mimic the training data. Second, we propose an optimization for efficient training and inference, which enables scaling to longer videos. Our approach shows excellent performance compared to both offline and online methods, and uses 20\% less compute. The annotations produced are much more comprehensive and frequent, and can further be utilized in automatic video tagging and in large-scale video data harvesting.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
