Traditional temporal action detection (TAD) usually handles untrimmed videos with small number of action instances from a single label (e.g., ActivityNet, THUMOS). However, this setting might be unrealistic as different classes of actions often co-occur in practice. In this paper, we focus on the task of multi-label temporal action detection that aims to localize all action instances from a multi-label untrimmed video. Multi-label TAD is more challenging as it requires for fine-grained class discrimination within a single video and precise localization of the co-occurring instances. To mitigate this issue, we extend the sparse query-based detection paradigm from the traditional TAD and propose the multi-label TAD framework of PointTAD. Specifically, our PointTAD introduces a small set of learnable query points to represent the important frames of each action instance. This point-based representation provides a flexible mechanism to localize the discriminative frames at boundaries and as well the important frames inside the action. Moreover, we perform the action decoding process with the Multi-level Interactive Module to capture both point-level and instance-level action semantics. Finally, our PointTAD employs an end-to-end trainable framework simply based on RGB input for easy deployment. We evaluate our proposed method on two popular benchmarks and introduce the new metric of detection-mAP for multi-label TAD. Our model outperforms all previous methods by a large margin under the detection-mAP metric, and also achieves promising results under the segmentation-mAP metric. Code is available at https://github.com/MCG-NJU/PointTAD.
Micro-gesture Online Recognition using Learnable Query Points
School of Computer Science and Information Engineering(计算机与信息工程学院);School of Artificial Intelligence(人工智能学院);Hefei University of Technology (HFUT)(合肥工业大学);Key Laboratory of Knowledge Engineering with Big Data (HFUT), Ministry of Education(大数据知识工程教育部重点实验室);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院);Zhejiang University(浙江大学);Anhui Zhonghuitong Technology Co., Ltd.(安徽中汇通科技有限公司)
ViGT: Proposal-free Video Grounding with Learnable Token in Transformer
Hefei University of Technology(合肥工业大学);Key Laboratory of Knowledge Engineering with Big Data, Ministry of Education(教育部大数据知识工程重点实验室);Intelligent Interconnected Systems Laboratory of Anhui Province(安徽省智能互联系统实验室);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院)
MS-DETR: Natural Language Video Localization with Sampling Moment-Moment Interaction
Nanyang Technological University(南洋理工大学);Institute for Infocomm Research, A*STAR(新加坡科技研究局资讯通信研究院);Centre for Frontier AI Research, A*STAR(新加坡科技研究局前沿人工智能研究中心)
Video Referring Expression Comprehension via Transformer with Content-conditioned Query
Peking University(北京大学);International Digital Economy Academy (IDEA)(国际数字经济学院(IDEA));Hong Kong University of Science and Technology(香港科技大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
SGIFormer: Semantic-guided and Geometric-enhanced Interleaving Transformer for 3D Instance Segmentation
Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University(电子工程系,香港理工大学);School of Mechanical Science and Engineering, Huazhong University of Science and Technology(机械科学与工程学院,华中科技大学)
Adaptive Query Prompting for Multi-Domain Landmark Detection
School of Computer Science and Technology, Guangdong University of Technology(广东技术大学计算机科学与技术学院);School of Automation, Guangdong University of Technology(广东技术大学自动化学院);Faculty of Applied Sciences, Macao Polytechnic University(澳门 polytechnic university 应用科学学院);School of Information Science and Technology, Guangdong University of Foreign Studies(广东外语外贸大学信息科学与技术学院);School of Computer Science and Engineering, Huizhou University(惠州市大学计算机科学与工程学院)
PaQ-DETR: Learning Pattern and Quality-Aware Dynamic Queries for Object Detection
New York University(纽约大学);Boise State University(博伊西州立大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of California, Irvine(加州大学 Irvine 分校);Illinois Institute of Technology(伊利诺伊理工学院);University of Pittsburgh(匹兹堡大学)
CRISP: Contrastive Residual Injection and Semantic Prompting for Continual Video Instance Segmentation
State Key Laboratory of Robotics and Intelligent Systems, Shenyang Institute of Automation, Chinese Academy of Sciences(机器人与智能系统国家重点实验室,沈阳自动化研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
StereoMV2D: A Sparse Temporal Stereo-Enhanced Framework for Robust Multi-View 3D Object Detection
the school of automation, Northwestern Polytechnical University(自动化学院,西北工业大学);the school of electronic and information engineering, Suzhou University of Science and Technology(电子信息工程学院,苏州科技大学)
Enhancing Partially Relevant Video Retrieval with Robust Alignment Learning
University of Science and Technology of China(中国科学技术大学);MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China(中国科学技术大学脑启发式智能感知与认知实验室);Zhejiang Gongshang University(浙江工商大学);ReLER, CCAI, Zhejiang University(ReLER,中国计算机学会,浙江大学)
RatioWaveNet: A Learnable RDWT Front-End for Robust and Interpretable EEG Motor-Imagery Classification
Department of Electrical, Electronics and Informatics Engineering, University of Catania(电子、电子与信息工程系,卡塔尼亚大学);Department of Engineering, University of Palermo(工程系,巴勒莫大学)
Multi-proposal Collaboration and Multi-task Training for Weakly-supervised Video Moment Retrieval
College of Computer Science and Electronic Engineering(计算机科学与电子工程学院);Faculty of Electronic Engineering and Computer Science(电子工程与计算机科学系);Mathematical and Data Science Center(数学与数据科学中心);Graduate School of Informatics(信息科学研究生院)
Multi-Grained Query-Guided Set Prediction Network for Grounded Multimodal Named Entity Recognition
School of Artificial Intelligence, Sun Yat-sen University(中山大学人工智能学院);School of Atmospheric Sciences, Sun Yat-sen University(中山大学大气科学学院);School of Electronics and Communication Engineering, Sun Yat-sen University(中山大学电子与通信工程学院);State Key Laboratory of Intelligent Game, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所智能游戏国家重点实验室);Pazhou Lab(琶洲实验室)