实时视频字幕生成
Live Video Captioning
AI总结：
本文提出实时视频字幕生成（LVC）新范式，结合可变形变换器与时间滤波在线生成字幕，并设计新评估指标，在ActivityNet Captions上超越离线方法。
摘要：
密集视频字幕生成涉及检测和描述视频序列中的事件。传统方法在离线设置下运行，假设整个视频可用于分析。相比之下，在这项工作中，我们引入了一种开创性的范式：实时视频字幕生成（LVC），其中必须在线方式为视频流生成字幕。这一转变带来了独特的挑战，包括处理事件的局部观察以及对动作的时间预期需求。我们正式定义了LVC的新问题，并提出了专门为这种在线场景设计的创新评估指标，展示了它们相对于传统指标的优势。为了解决LVC的新复杂性，我们提出了一种新模型，该模型结合了可变形变换器与时间滤波，从而能够对视频流进行有效的字幕生成。在ActivityNet Captions数据集上进行的大量实验验证了所提出的方法，展示了其在LVC设置中相对于最先进的离线方法的优越性能。为了促进进一步研究，我们提供了我们的模型结果以及集成了新指标的评估工具包，网址为：https://github.com/gramuah/lvc。
英文摘要：
Dense video captioning involves detecting and describing events within video sequences. Traditional methods operate in an offline setting, assuming the entire video is available for analysis. In contrast, in this work we introduce a groundbreaking paradigm: Live Video Captioning (LVC), where captions must be generated for video streams in an online manner. This shift brings unique challenges, including processing partial observations of the events and the need for a temporal anticipation of the actions. We formally define the novel problem of LVC and propose innovative evaluation metrics specifically designed for this online scenario, demonstrating their advantages over traditional metrics. To address the novel complexities of LVC, we present a new model that combines deformable transformers with temporal filtering, enabling effective captioning over video streams. Extensive experiments on the ActivityNet Captions dataset validate the proposed approach, showcasing its superior performance in the LVC setting compared to state-of-the-art offline methods. To foster further research, we provide the results of our model and an evaluation toolkit with the new metrics integrated at: https://github.com/gramuah/lvc.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
