arXiv 2506.160822025-06-19cs.CV

PR-DETR：为密集视频描述注入位置与关系先验

PR-DETR: Injecting Position and Relation Prior for Dense Video Captioning

National Key Laboratory of Human-Machine Hybrid Augmented Intelligence（人机混合增强智能国家重点实验室）；National Engineering Research Center for Visual Information and Applications（视觉信息与应用国家工程研究中心）；Institute of Artificial Intelligence and Robotics（人工智能与机器人研究院）；Xi’an Jiaotong University（西安交通大学）

Yizhe Li, Sanping Zhou, Zheng Qin, Le Wang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

PR-DETR通过向检测Transformer注入显式位置与关系先验，提升密集视频描述中事件定位精度与描述质量，在ActivityNet Captions和YouCook2上表现优异。

摘要：

密集视频描述是一项具有挑战性的任务，旨在对未剪辑视频中的多个事件进行定位并生成描述。近期研究主要采用基于Transformer的架构，以端到端的方式联合执行两个子任务，即事件定位和描述生成。基于检测Transformer的一般理念，这些方法隐式地学习事件位置和事件语义，这需要大量的训练数据，并限制了模型在实际中的性能。在本文中，我们提出了一种新颖的密集视频描述框架，名为PR-DETR，该框架将显式的位置和关系先验注入检测Transformer中，以同时提高定位精度和描述质量。一方面，我们首先生成一组位置锚定查询，以提供关于潜在事件的场景特定位置和语义信息作为位置先验，这些查询作为初始事件搜索区域，以消除不合理的事件提议。另一方面，我们进一步设计了一个事件关系编码器，以显式计算事件边界之间的关系作为关系先验，以指导事件交互，从而提高描述的语义连贯性。我们进行了广泛的消融研究，以验证位置和关系先验的有效性。实验结果还表明，我们的方法在ActivityNet Captions和YouCook2数据集上取得了有竞争力的性能。

英文摘要：

Dense video captioning is a challenging task that aims to localize and caption multiple events in an untrimmed video. Recent studies mainly follow the transformer-based architecture to jointly perform the two sub-tasks, i.e., event localization and caption generation, in an end-to-end manner. Based on the general philosophy of detection transformer, these methods implicitly learn the event locations and event semantics, which requires a large amount of training data and limits the model's performance in practice. In this paper, we propose a novel dense video captioning framework, named PR-DETR, which injects the explicit position and relation prior into the detection transformer to improve the localization accuracy and caption quality, simultaneously. On the one hand, we first generate a set of position-anchored queries to provide the scene-specific position and semantic information about potential events as position prior, which serves as the initial event search regions to eliminate the implausible event proposals. On the other hand, we further design an event relation encoder to explicitly calculate the relationship between event boundaries as relation prior to guide the event interaction to improve the semantic coherence of the captions. Extensive ablation studies are conducted to verify the effectiveness of the position and relation prior. Experimental results also show the competitive performance of our method on ActivityNet Captions and YouCook2 datasets.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。