arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.20892cs.CVcs.AI
收藏

MALT:用于在线动作检测的多尺度动作学习Transformer

MALT: Multi-scale Action Learning Transformer for Online Action Detection

Southeast University(东南大学);Southeast University Shenzhen Research Institute(东南大学深圳研究院)

Zhipeng Yang, Ruoyu Wang, Yang Tan, Liping Xie

AI总结:

针对在线动作检测中多尺度特征提取问题,提出多尺度动作学习Transformer(MALT),通过分层编码器、循环解码器和稀疏注意力帧评分机制,在THUMOS'14和TVSeries数据集上达到最先进性能。

摘要:

在线动作检测(OAD)旨在从流媒体视频中实时识别正在进行的动作,且无法访问未来帧。由于这些动作表现出不同的粒度尺度,从粗略到精细,将整个动作帧集合投影到单一潜在编码可能会导致局部信息缺失,因此需要获取多个尺度的动作特征。在本文中,我们提出了一种多尺度动作学习transformer(MALT),它包含一个新颖的循环解码器(用于特征融合),该解码器参数更少且训练效率更高。进一步提出了一种具有多个编码分支的分层编码器,以捕获多尺度动作特征。前一个分支的输出随后作为交叉注意力计算的一部分增量输入到后一个分支。通过这种方式,随着分支加深,输出特征从粗略过渡到精细。我们还引入了一种采用稀疏注意力的显式帧评分机制,该机制能更高效地过滤无关帧,而无需额外的网络。所提出的方法在两个基准数据集(THUMOS'14和TVSeries)上达到了最先进的性能,优于所有用于比较的现有模型,在THUMOS'14上的mAP为0.2%,在TVseries上的mcAP为0.1%。

英文摘要:

Online action detection (OAD) aims to identify ongoing actions from streaming video in real-time, without access to future frames. Since these actions manifest at varying scales of granularity, ranging from coarse to fine, projecting an entire set of action frames to a single latent encoding may result in a lack of local information, necessitating the acquisition of action features across multiple scales. In this paper, we propose a multi-scale action learning transformer (MALT), which includes a novel recurrent decoder (used for feature fusion) that includes fewer parameters and can be trained more efficiently. A hierarchical encoder with multiple encoding branches is further proposed to capture multi-scale action features. The output from the preceding branch is then incrementally input to the subsequent branch as part of a cross-attention calculation. In this way, output features transition from coarse to fine as the branches deepen. We also introduce an explicit frame scoring mechanism employing sparse attention, which filters irrelevant frames more efficiently, without requiring an additional network. The proposed method achieved state-of-the-art performance on two benchmark datasets (THUMOS'14 and TVSeries), outperforming all existing models used for comparison, with an mAP of 0.2% for THUMOS'14 and an mcAP of 0.1% for TVseries.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑