进度感知的视频帧字幕生成
Progress-Aware Video Frame Captioning
AI总结：
本文提出帧级进度感知视频字幕生成任务，构建ProgressCaptioner模型及FrameCap数据集和FrameCapEval基准，显著提升字幕时间精度，并应用于关键帧选择与视频理解。
摘要：
虽然图像字幕为单个图像提供孤立的描述，视频字幕为整个视频片段提供单一叙述，但我们的工作探索了一个重要的中间地带：帧级别的进度感知视频字幕生成。这一新任务旨在生成时间上细粒度的字幕，不仅准确描述每一帧，还能捕捉整个视频序列中动作的细微进展。尽管现有领先的视觉语言模型具有强大的能力，但它们往往难以辨别帧间差异的细微之处。为了解决这个问题，我们提出了ProgressCaptioner，一种旨在捕捉动作序列中细粒度时间动态的字幕生成模型。同时，我们开发了FrameCap数据集以支持训练，以及FrameCapEval基准来评估字幕质量。结果表明，ProgressCaptioner显著超越了领先的字幕生成模型，生成了准确捕捉动作进展的精确字幕，并为视频字幕的时间精度设立了新标准。最后，我们展示了我们方法的实际应用，特别是在辅助关键帧选择和推进视频理解方面，突显了其广泛的实用性。
英文摘要：
While image captioning provides isolated descriptions for individual images, and video captioning offers one single narrative for an entire video clip, our work explores an important middle ground: progress-aware video captioning at the frame level. This novel task aims to generate temporally fine-grained captions that not only accurately describe each frame but also capture the subtle progression of actions throughout a video sequence. Despite the strong capabilities of existing leading vision language models, they often struggle to discern the nuances of frame-wise differences. To address this, we propose ProgressCaptioner, a captioning model designed to capture the fine-grained temporal dynamics within an action sequence. Alongside, we develop the FrameCap dataset to support training and the FrameCapEval benchmark to assess caption quality. The results demonstrate that ProgressCaptioner significantly surpasses leading captioning models, producing precise captions that accurately capture action progression and set a new standard for temporal precision in video captioning. Finally, we showcase practical applications of our approach, specifically in aiding keyframe selection and advancing video understanding, highlighting its broad utility.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
