用于多标签时序动作检测的双重DETR
Dual DETRs for Multi-Label Temporal Action Detection
AI总结：
本文提出DualDETR框架，通过实例级与边界级双分支查询解码及联合查询初始化，提升多标签时序动作检测的边界定位性能。
摘要：
时序动作检测（TAD）旨在识别未修剪视频中的动作边界和相应类别。受DETR在目标检测中成功的启发，一些方法已将基于查询的框架应用于TAD任务。然而，这些方法主要遵循DETR在实例级别预测动作（即通过其中心点识别每个动作），导致边界定位不够理想。为解决这一问题，我们提出了一种新的双级别基于查询的TAD框架，即DualDETR，从实例级别和边界级别同时检测动作。在不同级别进行解码需要不同粒度的语义，因此我们引入了一种双分支解码结构。该结构为不同级别构建了独特的解码过程，有助于在每个级别显式捕获时间线索和语义。在双分支设计的基础上，我们提出了一种联合查询初始化策略来对齐两个级别的查询。具体而言，我们利用编码器提议以一对一的方式匹配每个级别的查询。然后，使用来自匹配动作提议的位置和内容先验来初始化匹配的查询。对齐的双级别查询可以在后续解码过程中利用互补线索细化匹配的提议。我们在三个具有挑战性的多标签TAD基准上评估了DualDETR。实验结果表明，DualDETR优于现有的最先进方法，在det-mAP下取得了显著改进，并在seg-mAP下提供了令人印象深刻的结果。
英文摘要：
Temporal Action Detection (TAD) aims to identify the action boundaries and the corresponding category within untrimmed videos. Inspired by the success of DETR in object detection, several methods have adapted the query-based framework to the TAD task. However, these approaches primarily followed DETR to predict actions at the instance level (i.e., identify each action by its center point), leading to sub-optimal boundary localization. To address this issue, we propose a new Dual-level query-based TAD framework, namely DualDETR, to detect actions from both instance-level and boundary-level. Decoding at different levels requires semantics of different granularity, therefore we introduce a two-branch decoding structure. This structure builds distinctive decoding processes for different levels, facilitating explicit capture of temporal cues and semantics at each level. On top of the two-branch design, we present a joint query initialization strategy to align queries from both levels. Specifically, we leverage encoder proposals to match queries from each level in a one-to-one manner. Then, the matched queries are initialized using position and content prior from the matched action proposal. The aligned dual-level queries can refine the matched proposal with complementary cues during subsequent decoding. We evaluate DualDETR on three challenging multi-label TAD benchmarks. The experimental results demonstrate the superior performance of DualDETR to the existing state-of-the-art methods, achieving a substantial improvement under det-mAP and delivering impressive results under seg-mAP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
