arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.16729cs.CV
收藏

用于时序动作检测的预测反馈 DETR

Prediction-Feedback DETR for Temporal Action Detection

Sungkyunkwan University(成均馆大学)

Jihwan Kim, Miso Lee, Cheol-Ho Cho, Jihyun Lee, Jae-Pil Heo

AI总结:

本文针对基于DETR的时序动作检测中交叉注意力的坍缩及捷径问题,提出Pred-DETR框架,利用预测反馈目标对齐注意力与预测,在多个基准上取得最先进性能。

摘要:

时序动作检测(TAD)对于真实世界视频应用而言是基础且具挑战性的。利用 Transformer 的独特优势,各种基于 DETR 的方法已被应用于 TAD。然而,近期研究发现自注意力中的注意力坍缩会导致 TAD 中 DETR 的性能退化。基于先前研究,本文新近探讨了基于 DETR 的 TAD 方法中交叉注意力的注意力坍缩问题。此外,我们的发现表明,交叉注意力展现出与预测不同的模式,指示了一种捷径现象。为解决此问题,我们提出了一个新框架 Prediction-Feedback DETR (Pred-DETR),它利用预测来恢复坍缩,并使交叉和自注意力与预测对齐。具体而言,我们利用预测关系的指导设计了新颖的预测反馈目标。最终,Pred-DETR 显著缓解了坍缩,并在包括 THUMOS14、ActivityNet-v1.3、HACS 和 FineAction 在内的多个挑战性基准上,于基于 DETR 的方法中取得了最先进的性能。

英文摘要:

Temporal Action Detection (TAD) is fundamental yet challenging for real-world video applications. Leveraging the unique benefits of transformers, various DETR-based approaches have been adopted in TAD. However, it has recently been identified that the attention collapse in self-attention causes the performance degradation of DETR for TAD. Building upon previous research, this paper newly addresses the attention collapse problem in cross-attention within DETR-based TAD methods. Moreover, our findings reveal that cross-attention exhibits patterns distinct from predictions, indicating a short-cut phenomenon. To resolve this, we propose a new framework, Prediction-Feedback DETR (Pred-DETR), which utilizes predictions to restore the collapse and align the cross- and self-attention with predictions. Specifically, we devise novel prediction-feedback objectives using guidance from the relations of the predictions. As a result, Pred-DETR significantly alleviates the collapse and achieves state-of-the-art performance among DETR-based methods on various challenging benchmarks including THUMOS14, ActivityNet-v1.3, HACS, and FineAction.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑