arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.06187cs.CV
收藏

一种用于密集多标签动作检测的高效且有效的方法

An Effective-Efficient Approach for Dense Multi-Label Action Detection

University of Surrey(萨里大学)

Faegheh Sardari, Armin Mustafa, Philip J. B. Jackson, Adrian Hilton

AI总结:

本文针对密集多标签动作检测中时间位置信息丢失和动作共现关系建模计算开销大的问题,提出一种采用非分层结构和相对位置编码的Transformer网络,并通过新型学习范式在推理阶段无额外计算成本地建模动作共现依赖,在两个基准数据集上取得最优结果。

摘要:

与稀疏标签动作检测任务(视频的每个时间戳中只发生一个动作)不同,在密集多标签场景中,动作可以相互重叠。为了解决这一具有挑战性的任务,需要同时学习(i)时间依赖关系和(ii)动作共现关系。近期方法通过基于分层Transformer的网络提取多尺度特征来建模时间信息。然而,Transformer中的自注意力机制本质上会丢失时间位置信息。我们认为,将其与分层设计中的多次子采样过程相结合,会导致位置信息的进一步丢失。保留这些信息对于准确的动作检测至关重要。在本文中,我们通过提出一种新颖的基于Transformer的网络来解决这一问题,该网络(a)在建模不同范围的时间依赖关系时采用非分层结构,(b)在其Transformer层中嵌入相对位置编码。此外,为了建模动作共现关系,当前方法将类别关系显式嵌入到Transformer网络中。然而,这些方法在计算上并不高效,因为网络需要计算所有可能的动作类别对关系。我们还通过引入一种新颖的学习范式来克服这一挑战,该范式使网络能够从显式建模时间动作共现依赖中受益,而无需在推理期间增加额外的计算成本。我们在两个具有挑战性的密集多标签基准数据集上评估了我们所提出方法的性能,并表明我们的方法改进了当前最先进的结果。

英文摘要:

Unlike the sparse label action detection task, where a single action occurs in each timestamp of a video, in a dense multi-label scenario, actions can overlap. To address this challenging task, it is necessary to simultaneously learn (i) temporal dependencies and (ii) co-occurrence action relationships. Recent approaches model temporal information by extracting multi-scale features through hierarchical transformer-based networks. However, the self-attention mechanism in transformers inherently loses temporal positional information. We argue that combining this with multiple sub-sampling processes in hierarchical designs can lead to further loss of positional information. Preserving this information is essential for accurate action detection. In this paper, we address this issue by proposing a novel transformer-based network that (a) employs a non-hierarchical structure when modelling different ranges of temporal dependencies and (b) embeds relative positional encoding in its transformer layers. Furthermore, to model co-occurrence action relationships, current methods explicitly embed class relations into the transformer network. However, these approaches are not computationally efficient, as the network needs to compute all possible pair action class relations. We also overcome this challenge by introducing a novel learning paradigm that allows the network to benefit from explicitly modelling temporal co-occurrence action dependencies without imposing their additional computational costs during inference. We evaluate the performance of our proposed approach on two challenging dense multi-label benchmark datasets and show that our method improves the current state-of-the-art results.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑