Streaming video clips with large-scale video tokens impede vision transformers (ViTs) for efficient recognition, especially in video action detection where sufficient spatiotemporal representations are required for precise actor identification. In this work, we propose an end-to-end framework for efficient video action detection (EVAD) based on vanilla ViTs. Our EVAD consists of two specialized designs for video action detection. First, we propose a spatiotemporal token dropout from a keyframe-centric perspective. In a video clip, we maintain all tokens from its keyframe, preserve tokens relevant to actor motions from other frames, and drop out the remaining tokens in this clip. Second, we refine scene context by leveraging remaining tokens for better recognizing actor identities. The region of interest (RoI) in our action detector is expanded into temporal domain. The captured spatiotemporal actor identity representations are refined via scene context in a decoder with the attention mechanism. These two designs make our EVAD efficient while maintaining accuracy, which is validated on three benchmark datasets (i.e., AVA, UCF101-24, JHMDB). Compared to the vanilla ViT backbone, our EVAD reduces the overall GFLOPs by 43% and improves real-time inference speed by 40% with no performance degradation. Moreover, even at similar computational costs, our EVAD can improve the performance by 1.1 mAP with higher resolution inputs. Code is available at https://github.com/MCG-NJU/EVAD.
Beyond Overfitting: Doubly Adaptive Dropout for Generalizable AU Detection
Southeast University(东南大学);Nanjing University of Science and Technology(南京理工大学);Beijing Institute for General Artificial Intelligence(北京通用人工智能研究院);Nanyang Technological University(南洋理工大学)
STAR-Pro: Stage-Wise Token Adaptive Reduction with Progressive Refinement for Efficient Large Vision-Language Models
Peking University(北京大学);Nanyang Technological University, Singapore(南洋理工大学);University of Electronic Science and Technology of China(电子科技大学);University of Michigan, Ann Arbor(密歇根大学安娜堡分校);De Artificial Intelligence Lab(De人工智能实验室)
Improving Adversarial Transferability on Vision Transformers via Forward Propagation Refinement
Xi’an Jiaotong University(西安交通大学);Information Engineering University(信息工程大学);Nanjing University of Aeronautics and Astronautics(南京航空航天大学);Zhejiang Lab(之江实验室)
DFM-VLA: Iterative Action Refinement for Robot Manipulation via Discrete Flow Matching
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Harbin Institute of Technology(哈尔滨工业大学);ShanghaiTech University(上海科技大学);Shanghai Institute of Technical Physics, CAS(中国科学院上海技术物理研究所)
DRRG: A Discrete Diffusion Framework for Radiology Report Generation
School of Electrical and Computer Engineering, The University of Sydney(悉尼大学电气与计算机工程学院);Department of Radiology, The First Affiliated Hospital of Soochow University(苏州大学附属第一医院放射科);School of Computer Science, The University of Adelaide(阿德莱德大学计算机学院);School of Computing and Information Technology, University of Wollongong(卧龙岗大学计算与信息技术学院)
Certainty Is Redundant: Token Sparsification for Efficient Camouflaged Object Detection with Vision Foundation Models
VCIP, College of Computer Science, Nankai University(VCIP,计算机科学学院,南开大学);Academy for Advanced Interdisciplinary Studies, Nankai University(先进交叉学科研究院,南开大学);School of Computer Science and Engineering, Tianjin University of Technology(计算机科学与工程学院,天津工业大学);School of Information and Communication Engineering, UESTC(信息与通信工程学院,电子科技大学);Nankai International Advanced Research Institute, Shenzhen Futian(南开国际先进研究院,深圳福田)
NeuroDETR: Multiscale Discrete Spatiotemporal Tokenization for Asynchronous EEG Event Detection
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Zhongguancun Academy(中关村学院);Wuhan University(武汉大学)
Training-Free Hidden-State Refinement for Flow-Matching Image Generators
Huazhong University of Science and Technology(华中科技大学);Tongji University(同济大学);King Abdullah University of Science and Technology(阿卜杜拉国王科技大学);University of California, Merced(加州大学默塞德分校);University of Amsterdam(阿姆斯特丹大学)