Temporal Action Detection (TAD) is fundamental yet challenging for real-world video applications. Leveraging the unique benefits of transformers, various DETR-based approaches have been adopted in TAD. However, it has recently been identified that the attention collapse in self-attention causes the performance degradation of DETR for TAD. Building upon previous research, this paper newly addresses the attention collapse problem in cross-attention within DETR-based TAD methods. Moreover, our findings reveal that cross-attention exhibits patterns distinct from predictions, indicating a short-cut phenomenon. To resolve this, we propose a new framework, Prediction-Feedback DETR (Pred-DETR), which utilizes predictions to restore the collapse and align the cross- and self-attention with predictions. Specifically, we devise novel prediction-feedback objectives using guidance from the relations of the predictions. As a result, Pred-DETR significantly alleviates the collapse and achieves state-of-the-art performance among DETR-based methods on various challenging benchmarks including THUMOS14, ActivityNet-v1.3, HACS, and FineAction.
End-to-End Video Object Detection with Spatial-Temporal Transformers
Shanghai Jiao Tong University(上海交通大学);Qingyuan Research Institute, SJTU(上海交通大学清源研究院);Peking University(北京大学);Sensetime Research(商汤科技研究院);UCLA(加州大学洛杉矶分校)
MI-DETR: A Strong Baseline for Moving Infrared Small Target Detection with Motion Integration
University of Chinese Academy of Sciences(中国科学院大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);ShanghaiTech University(上海科技大学);Beijing Institute of Basic Medical Sciences(北京基础医学研究所)
GIRL-DETR: Gradient-Isolated Reinforcement Learning for Video Moment Retrieval
College of Electronics and Information Engineering, Sichuan University(四川大学电子信息工程学院);School of Intelligence Science and Technology, Nanjing University(南京大学智能科学与技术学院)
FeedbackSTS-Det: Sparse Frames-Based Spatio-Temporal Semantic Feedback Network for Moving Infrared Small Target Detection
School of Information and Communication Engineering and the Laboratory of Imaging Detection and Intelligent Perception, University of Electronic Science and Technology of China(电子科技大学信息与通信工程学院及成像探测与智能感知实验室);School of Television and Audiovisual Arts, Communication University of Zhejiang(浙江传媒学院电视与视听艺术学院)
Feedback Graph Convolutional Network for Skeleton-based Action Recognition
R&D Center of Artificial Intelligent, NUCTECH Company Limited(NUCTECH有限公司人工智能研发中心);Department of Engineering Physics, Tsinghua University(清华大学工程物理系);Informatics Institute, University of Amsterdam(阿姆斯特丹大学信息学院);Department of Computer Science and Information Systems, Birkbeck College(伦敦大学伯贝克学院计算机科学与信息系统系)
TAPTR: Tracking Any Point with Transformers as Detection
South China University of Technology(华南理工大学);International Digital Economy Academy (IDEA)(国际数字经济学院);The Hong Kong University of Science and Technology(香港科技大学);Tsinghua University(清华大学)
STF: Shallow-Level Temporal Feedback to Enhance Spiking Transformers
Zhejiang University(浙江大学);Westlake University(西湖大学);Nanjing University(南京大学);Donghua University(东华大学);Fudan University(复旦大学);University of Electronic Science and Technology of China(电子科技大学);Peking University(北京大学)