通过场景图整合实现长视频的细粒度字幕生成
Fine-Grained Captioning of Long Videos through Scene Graph Consolidation
AI总结：
本文提出一种基于场景图整合的长视频字幕生成框架，通过将片段级字幕解析为场景图并整合为统一表示，再经轻量级图到文本解码器生成视频级字幕，无需微调即可实现强零样本性能并显著降低计算成本。
摘要：
视觉-语言模型的最新进展在图像和短视频片段的字幕生成方面取得了令人瞩目的进步。然而，这些模型仍然受限于其有限的时间感受野，使得为长视频生成连贯且全面的字幕变得困难。尽管已经提出了若干方法来聚合视频片段之间的信息，但它们通常依赖监督微调或带来显著的计算开销。为应对这些挑战，我们提出了一种基于图整合的长视频字幕生成新框架。我们的方法首先使用现成的视觉字幕模型生成片段级字幕，对应于单个帧或短视频区间。随后，这些字幕被解析为独立的场景图，再被整合成一个统一的图表示，该表示在整段视频中同时保留整体上下文和细粒度细节。接着，一个轻量级的图到文本解码器生成最终的视频级字幕。该框架有效扩展了现有模型的时间理解能力，而无需在长视频数据集上进行任何额外微调。实验结果表明，我们的方法显著优于现有的基于大语言模型的整合方法，在实现强大零样本性能的同时大幅降低了计算成本。
英文摘要：
Recent advances in vision-language models have led to impressive progress in caption generation for images and short video clips. However, these models remain constrained by their limited temporal receptive fields, making it difficult to produce coherent and comprehensive captions for long videos. While several methods have been proposed to aggregate information across video segments, they often rely on supervised fine-tuning or incur significant computational overhead. To address these challenges, we introduce a novel framework for long video captioning based on graph consolidation. Our approach first generates segment-level captions, corresponding to individual frames or short video intervals, using off-the-shelf visual captioning models. These captions are then parsed into individual scene graphs, which are subsequently consolidated into a unified graph representation that preserves both holistic context and fine-grained details throughout the video. A lightweight graph-to-text decoder then produces the final video-level caption. This framework effectively extends the temporal understanding capabilities of existing models without requiring any additional fine-tuning on long video datasets. Experimental results show that our method significantly outperforms existing LLM-based consolidation approaches, achieving strong zero-shot performance while substantially reducing computational costs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
