An audio-visual event (AVE) is denoted by the correspondence of the visual and auditory signals in a video segment. Precise localization of the AVEs is very challenging since it demands effective multi-modal feature correspondence to ground the short and long range temporal interactions. Existing approaches struggle in capturing the different scales of multi-modal interaction due to ineffective multi-modal training strategies. To overcome this limitation, we introduce AVE-CLIP, a novel framework that integrates the AudioCLIP pre-trained on large-scale audio-visual data with a multi-window temporal transformer to effectively operate on different temporal scales of video frames. Our contributions are three-fold: (1) We introduce a multi-stage training framework to incorporate AudioCLIP pre-trained with audio-image pairs into the AVE localization task on video frames through contrastive fine-tuning, effective mean video feature extraction, and multi-scale training phases. (2) We propose a multi-domain attention mechanism that operates on both temporal and feature domains over varying timescales to fuse the local and global feature variations. (3) We introduce a temporal refining scheme with event-guided attention followed by a simple-yet-effective post processing step to handle significant variations of the background over diverse events. Our method achieves state-of-the-art performance on the publicly available AVE dataset with 5.9% mean accuracy improvement which proves its superiority over existing approaches.
Bridging the Task Gap: Multi-Task Adversarial Transferability in CLIP and Its Derivatives
Shenzhen Campus of Sun Yat-sen University, China(中山大学深圳校区);The National University of Singapore, Singapore(新加坡国立大学);National Key Laboratory of Science and Technology on Information System Security, China(信息系统安全科学技术国家重点实验室)
School of Cyber Science and Technology Sun Yat-Sen University(中山大学网络空间安全学院);Department of Computer Science National University of Singapore(新加坡国立大学计算机科学系)
HAC: Parameter-Efficient Hyperbolic Adaptation of CLIP for Zero-Shot VQA
Department of Computer Science, University of Verona(威尼斯大学计算机科学系);EVS - Embedded Vision Systems Srl(嵌入式视觉系统公司);AI for Good (AIGO), Istituto Italiano di Tecnologia(AI for Good(AIGO),意大利技术研究院)
Describe, Adapt and Combine: Empowering CLIP Encoders for Open-set 3D Object Retrieval
Huazhong Agricultural University(华中农业大学);Shenzhen University(深圳大学);The University of Hong Kong(香港大学);University of Louisville(路易斯安那大学);ByteDance(字节跳动);Huazhong University of Science and Technology(华中科技大学)
Indian Institute of Technology Hyderabad(印度理工学院海得拉巴分校);Indian Institute of Technology Madras(印度理工学院马德拉斯分校);Wadhwani School of Data Science & AI(瓦德瓦尼数据科学与人工智能学院)
Bridging Vision Foundation Model Priors with CLIP for Spatial-aware Few-shot Anomaly Detection in Medical Images
The Chinese University of Hong Kong(香港中文大学);The University of Hong Kong(香港大学);Institute of Medical Intelligence and XR, The Chinese University of Hong Kong(香港中文大学医学智能与扩展现实研究所)
RAVID: Retrieval-Augmented Visual Detection: A Knowledge-Driven Approach for AI-Generated Image Identification
Laboratory of IEMN, Univ. Polytechnique Hauts-de-France(IEMN实验室,法国高等技术法国大学);KU 6G Research Center, Khalifa University(KU 6G研究中心,哈利法大学);Sorbonne Center for Artificial Intelligence, Sorbonne University(人工智能研究中心,索邦大学)
Wavelet-Driven Generalizable Framework for Deepfake Face Forgery Detection
International Institute of Information Technology Hyderabad(印度信息技术学院海得拉巴分校);Indian Institute of Technology Hyderabad(印度理工学院海得拉巴分校);Michigan State University(密歇根州立大学)