arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.11035cs.CV
收藏

PointTAD:基于可学习查询点的多标签时态动作检测

PointTAD: Multi-Label Temporal Action Detection with Learnable Query Points

Nanjing University(南京大学);Tencent(腾讯);Shanghai AI Lab(上海人工智能实验室)

Jing Tan, Xiaotong Zhao, Xintian Shi, Bin Kang, Limin Wang

摘要:

传统的时间动作检测(Temporal Action Detection,简称TACT)通常处理来自单个标签(例如,ActivityNet,THUMOS).然而,这种设置可能不切实际,因为不同类别的行动在实践中往往同时发生。在本文中,我们专注于任务的多标签的时间动作检测,旨在本地化的所有动作实例从多标签的未修剪的视频。多标签分类更具挑战性,因为它需要在单个视频中进行细粒度的类别区分,并精确定位共现实例。为了解决这个问题,我们扩展了基于稀疏查询的检测范式,从传统的并行检测,并提出了多标签并行检测框架的PointData。具体来说,我们的PointSort引入了一小部分可学习的查询点来表示每个动作实例的重要框架。这种基于点的表示提供了一种灵活的机制来定位边界处的判别帧以及动作内部的重要帧。此外,我们执行的动作解码过程与多层次交互模块捕捉点级和实例级的动作语义。最后,我们的PointSide采用了一个端到端的可训练框架,简单地基于RGB输入,便于部署。我们评估我们提出的方法在两个流行的基准,并介绍了新的度量检测-mAP的多标签标签识别。我们的模型在检测-mAP度量下的性能优于所有以前的方法,并且在分割-mAP度量下也取得了有希望的结果。代码可在https://github.com/MCG-NJU/PointTAD上获得。

英文摘要:

Traditional temporal action detection (TAD) usually handles untrimmed videos with small number of action instances from a single label (e.g., ActivityNet, THUMOS). However, this setting might be unrealistic as different classes of actions often co-occur in practice. In this paper, we focus on the task of multi-label temporal action detection that aims to localize all action instances from a multi-label untrimmed video. Multi-label TAD is more challenging as it requires for fine-grained class discrimination within a single video and precise localization of the co-occurring instances. To mitigate this issue, we extend the sparse query-based detection paradigm from the traditional TAD and propose the multi-label TAD framework of PointTAD. Specifically, our PointTAD introduces a small set of learnable query points to represent the important frames of each action instance. This point-based representation provides a flexible mechanism to localize the discriminative frames at boundaries and as well the important frames inside the action. Moreover, we perform the action decoding process with the Multi-level Interactive Module to capture both point-level and instance-level action semantics. Finally, our PointTAD employs an end-to-end trainable framework simply based on RGB input for easy deployment. We evaluate our proposed method on two popular benchmarks and introduce the new metric of detection-mAP for multi-label TAD. Our model outperforms all previous methods by a large margin under the detection-mAP metric, and also achieves promising results under the segmentation-mAP metric. Code is available at https://github.com/MCG-NJU/PointTAD.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑