面向密集视频字幕生成的时间缩放状态空间模型
Time-Scaling State-Space Models for Dense Video Captioning
AI总结：
针对密集视频字幕生成中长视频处理的计算、内存限制及无法在线处理的问题，提出带转移状态的SSM方法，提升其长上下文状态维持能力，可在线生成字幕，适配长视频且FLOPs降低至1/7。
摘要：
密集视频字幕生成是一项具有挑战性的视频理解任务，其目标是同时将视频分割为一系列有意义的连续事件，并生成详细的字幕以准确描述每个事件。现有方法在处理密集视频字幕生成相关的长视频时，常因计算复杂度和内存限制遇到困难。此外，传统方法需要将完整视频作为输入才能生成结果，这使得视频无法进行在线处理。我们通过将状态空间模型（SSMs）进行时间缩放，使其能处理比以往更长的序列，来解决这些挑战。我们提出的方法——带转移状态的状态空间模型（State-Space Models with Transfer State），结合了SSMs的长序列处理能力和循环特性，解决了SSMs无法在极长上下文下维持状态的主要局限，有效进一步提升了SSMs的时间缩放能力。该模型特别适合以在线或流式的方式即时生成字幕，无需等待完整视频处理完毕，在实际应用中更具优势。当应用于密集视频字幕生成任务时，我们的方法能很好地适配不同视频长度，且浮点运算量（FLOPs）仅为原有方法的1/7。
英文摘要：
Dense video captioning is a challenging video understanding task which aims to simultaneously segment the video into a sequence of meaningful consecutive events and to generate detailed captions to accurately describe each event. Existing methods often encounter difficulties when working with the long videos associated with dense video captioning, due to the computational complexity and memory limitations. Furthermore, traditional approaches require the entire video as input, in order to produce an answer, which precludes online processing of the video. We address these challenges by time-scaling State-Space Models (SSMs) to even longer sequences than before. Our approach, State-Space Models with Transfer State, combines both the long-sequence and recurrent properties of SSMs and addresses the main limitation of SSMs which are otherwise not able to sustain their state for very long contexts, effectively scaling SSMs further in time. The proposed model is particularly suitable for generating captions on-the-fly, in an online or streaming manner, without having to wait for the full video to be processed, which is more beneficial in practice. When applied to dense video captioning, our approach scales well with video lengths and uses 7x fewer FLOPs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
