Early action prediction (EAP) aims to recognize human actions from a part of action execution in ongoing videos, which is an important task for many practical applications. Most prior works treat partial or full videos as a whole, ignoring rich action knowledge hidden in videos, i.e., semantic consistencies among different partial videos. In contrast, we partition original partial or full videos to form a new series of partial videos and mine the Action-Semantic Consistent Knowledge (ASCK) among these new partial videos evolving in arbitrary progress levels. Moreover, a novel Rich Action-semantic Consistent Knowledge network (RACK) under the teacher-student framework is proposed for EAP. Firstly, we use a two-stream pre-trained model to extract features of videos. Secondly, we treat the RGB or flow features of the partial videos as nodes and their action semantic consistencies as edges. Next, we build a bi-directional semantic graph for the teacher network and a single-directional semantic graph for the student network to model rich ASCK among partial videos. The MSE and MMD losses are incorporated as our distillation loss to enrich the ASCK of partial videos from the teacher to the student network. Finally, we obtain the final prediction by summering the logits of different subnetworks and applying a softmax layer. Extensive experiments and ablative studies have been conducted, demonstrating the effectiveness of modeling rich ASCK for EAP. With the proposed RACK, we have achieved state-of-the-art performance on three benchmarks. The code is available at https://github.com/lily2lab/RACK.git.
University of Moratuwa(莫拉图瓦大学);Mohamed Bin Zayed University of AI(穆罕默德·本·扎耶德人工智能大学);Australian National University(澳大利亚国立大学);Linköping University(林雪平大学)
Defect Spectrum: A Granular Look of Large-Scale Defect Datasets with Rich Semantics
Hong Kong University of Science and Technology, Guangzhou(香港科技大学(广州));HKUST(GZ) - SmartMore Joint Lab(香港科技大学(广州)- 赛摩智能联合实验室);SmartMore. Corp(赛摩智能公司);Hong Kong University of Science and Technology(香港科技大学)
Halt the Hallucination: Decoupling Signal and Semantic OOD Detection Based on Cascaded Early Rejection
School of Computer and Electronic Information, Nanjing Normal University, China(南京师范大学计算机与电子信息学院);School of Artificial Intelligence and Information Technology, Nanjing University of Chinese Medicine, China(南京中医药大学人工智能与信息技术学院)
Deep Binaries: Encoding Semantic-Rich Cues for Efficient Textual-Visual Cross Retrieval
School of Computing Sciences, University of East Anglia(东英吉利大学计算科学学院);Malong Technologies Co., Ltd.(码隆科技有限公司);Future Media Center, University of Electronic Science and Technology of China(电子科技大学未来媒体中心)
Free-VSC: Free Semantics from Visual Foundation Models for Unsupervised Video Semantic Compression
Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University(上海交通大学图像通信与网络工程研究所);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Deep learning for semantic segmentation of remote sensing images with rich spectral content
Univ. Savoie Mont Blanc, LISTIC(萨瓦勃朗峰大学 LISTIC 实验室);ONERA, The French Aerospace Lab(法国航空航天实验室 ONERA);Univ. Bretagne-Sud, UMR 6074, IRISA(南布列塔尼大学 IRISA 研究所)
Key Patch Proposer: Key Patches Contain Rich Information
Institute for AI Industry Research, Tsinghua University(清华大学人工智能产业研究院);Nanjing University of Aeronautics And Astronautics(南京航空航天大学);Tsinghua University(清华大学)
SAC-ViT: Semantic-Aware Clustering Vision Transformer with Early Exit
Faculty of Computing, Harbin Institute of Technology(哈尔滨工业大学计算机学院);Swiss Data Science Center(瑞士数据科学中心);School of Cyber Engineering, Xidian University(西安电子科技大学网络工程学院)