Stay in your Lane: 角色特定查询与重叠抑制损失用于密集视频描述
Stay in your Lane: Role Specific Queries with Overlap Suppression Loss for Dense Video Captioning
AI总结：
本文提出角色特定查询与重叠抑制损失，用于解决密集视频描述中的多任务干扰和时间冗余问题，通过分离定位与描述任务并增强语义一致性，提升描述精度。
摘要：
密集视频描述（DVC）是一个具有挑战性的多模态任务，涉及在视频中时间上定位多个事件并用自然语言描述它们。尽管基于查询的框架能够同时进行端到端的定位和描述处理，但其依赖共享查询往往导致两个任务之间的显著多任务干扰以及定位中的时间冗余。在本文中，我们提出利用角色特定查询将定位和描述分离为独立组件，使每个组件能够专门学习其角色。我们随后采用对比对齐来强制对应输出之间的语义一致性，确保在分离查询之间具有连贯的行为。此外，我们设计了一种新颖的抑制机制，其中查询之间的相互时间重叠会被惩罚以解决时间冗余，监督模型学习到不同的、不重叠的事件区域以实现更精确的定位。此外，我们引入了一个轻量级模块，捕捉核心事件概念，通过概念级表示进一步增强描述中的语义丰富性。我们通过在主要的DVC基准YouCook2和ActivityNet Captions上进行广泛的实验来证明我们方法的有效性。
英文摘要：
Dense Video Captioning (DVC) is a challenging multimodal task that involves temporally localizing multiple events within a video and describing them with natural language. While query-based frameworks enable the simultaneous, end-to-end processing of localization and captioning, their reliance on shared queries often leads to significant multi-task interference between the two tasks, as well as temporal redundancy in localization. In this paper, we propose utilizing role-specific queries that separate localization and captioning into independent components, allowing each to exclusively learn its role. We then employ contrastive alignment to enforce semantic consistency between the corresponding outputs, ensuring coherent behavior across the separated queries. Furthermore, we design a novel suppression mechanism in which mutual temporal overlaps across queries are penalized to tackle temporal redundancy, supervising the model to learn distinct, non-overlapping event regions for more precise localization. Additionally, we introduce a lightweight module that captures core event concepts to further enhance semantic richness in captions through concept-level representations. We demonstrate the effectiveness of our method through extensive experiments on major DVC benchmarks YouCook2 and ActivityNet Captions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
