Sali4Vid：面向密集视频字幕的显著性感知视频重加权与自适应字幕检索
Sali4Vid: Saliency-Aware Video Reweighting and Adaptive Caption Retrieval for Dense Video Captioning
AI总结：
针对现有密集视频字幕模型帧权重均等、忽略场景转换的问题，提出Sali4Vid框架，通过显著性感知视频重加权和自适应字幕检索优化任务效果，在两个基准上达SOTA。
摘要：
密集视频字幕旨在对视频中的事件进行时间定位，并为每个事件生成字幕。尽管近期研究提出了端到端模型，但它们存在两个局限：（1）仅对文本施加时间戳监督，却对所有视频帧一视同仁；（2）从固定大小的视频块中检索字幕，忽略了场景转换。为解决这些问题，我们提出Sali4Vid，这是一个简洁却有效的显著性感知框架。我们引入了显著性感知视频重加权模块，将时间戳标注转换为基于sigmoid的帧重要性权重；还引入了基于语义的自适应字幕检索模块，通过帧相似度对视频分段以捕捉场景转换，优化字幕检索效果。Sali4Vid在YouCook2和ViTT数据集上取得了当前最优结果，证明了协同改进视频加权与检索对密集视频字幕任务的增益。
英文摘要：
Dense video captioning aims to temporally localize events in video and generate captions for each event. While recent works propose end-to-end models, they suffer from two limitations: (1) applying timestamp supervision only to text while treating all video frames equally, and (2) retrieving captions from fixed-size video chunks, overlooking scene transitions. To address these, we propose Sali4Vid, a simple yet effective saliency-aware framework. We introduce Saliency-aware Video Reweighting, which converts timestamp annotations into sigmoid-based frame importance weights, and Semantic-based Adaptive Caption Retrieval, which segments videos by frame similarity to capture scene transitions and improve caption retrieval. Sali4Vid achieves state-of-the-art results on YouCook2 and ViTT, demonstrating the benefit of jointly improving video weighting and retrieval for dense video captioning
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
