arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.13152cs.CV
收藏

基于 Transformer 的时序动作检测长期预训练

Long-term Pre-training for Temporal Action Detection with Transformers

Sungkyunkwan University(成均馆大学)

Jihwan Kim, Miso Lee, Jae-Pil Heo

AI总结:

针对 Transformer 时序动作检测中的数据稀缺问题,提出包含类别级合成与长期预文本任务的 LTP,并在 ActivityNet-v1.3 和 THUMOS14 上大幅提升 DETR 类方法性能。

摘要:

时序动作检测(TAD)颇具挑战,但对真实世界视频应用至关重要。近来,用于 TAD 的基于 DETR 的模型因其独特优势而日益流行。然而,Transformer 需要庞大的数据集,不幸的是,TAD 中的数据稀缺会导致严重的性能退化。在本文中,我们识别出数据稀缺带来的两个关键问题:注意力坍缩和性能不均衡。为此,我们提出了一种为 Transformer 量身定制的新型预训练策略——长期预训练(LTP)。LTP 包含两个主要组成部分:1)类别级合成;2)长期预文本任务。首先,我们通过合并目标类别和非目标类别的视频片段合成长视频特征。尽管这些特征由裁剪数据创建,但它们类似于 TAD 中使用的未裁剪数据。此外,我们设计了两类长期预文本任务以学习长期依赖关系。它们施加长期条件,例如寻找第二至第四个动作或短时动作。大量实验表明,在 ActivityNet-v1.3 和 THUMOS14 上,该方法在基于 DETR 的方法中以大幅优势取得了最先进性能。此外,我们证明 LTP 显著缓解了 TAD 中的数据稀缺问题。

英文摘要:

Temporal action detection (TAD) is challenging, yet fundamental for real-world video applications. Recently, DETR-based models for TAD have been prevailing thanks to their unique benefits. However, transformers demand a huge dataset, and unfortunately data scarcity in TAD causes a severe degeneration. In this paper, we identify two crucial problems from data scarcity: attention collapse and imbalanced performance. To this end, we propose a new pre-training strategy, Long-Term Pre-training (LTP), tailored for transformers. LTP has two main components: 1) class-wise synthesis, 2) long-term pretext tasks. Firstly, we synthesize long-form video features by merging video snippets of a target class and non-target classes. They are analogous to untrimmed data used in TAD, despite being created from trimmed data. In addition, we devise two types of long-term pretext tasks to learn long-term dependency. They impose long-term conditions such as finding second-to-fourth or short-duration actions. Our extensive experiments show state-of-the-art performances in DETR-based methods on ActivityNet-v1.3 and THUMOS14 by a large margin. Moreover, we demonstrate that LTP significantly relieves the data scarcity issues in TAD.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑