We introduce a simple method that employs pre-trained CLIP encoders to enhance model generalization in the ALFRED task. In contrast to previous literature where CLIP replaces the visual encoder, we suggest using CLIP as an additional module through an auxiliary object detection objective. We validate our method on the recently proposed Episodic Transformer architecture and demonstrate that incorporating CLIP improves task performance on the unseen validation set. Additionally, our analysis results support that CLIP especially helps with leveraging object descriptions, detecting small objects, and interpreting rare words.
Spectral Prompt Tuning:Unveiling Unseen Classes for Zero-Shot Semantic Segmentation
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统全国重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
How can we imbue robots with the ability to manipulate objects precisely but also to reason about them in terms of abstract concepts? Recent works in manipulation have shown that end-to-end networks can learn dexterous skills that require precise spatial reasoning, but these methods often fail to generalize to new goals or quickly learn transferable concepts across tasks. In parallel, there has been great progress in learning generalizable semantic representations for vision and language by training on large-scale internet data, however these representations lack the spatial understanding nece…
Nanyang Technological University（南洋理工大学）；Shanghai AI Laboratory（上海人工智能实验室）
Contrastive Language-Image Pre-training (CLIP) has made a remarkable breakthrough in open-vocabulary zero-shot image recognition. Many recent studies leverage the pre-trained CLIP models for image-level classification and manipulation. In this paper, we wish examine the intrinsic potential of CLIP for pixel-level dense prediction, specifically in semantic segmentation. To this end, with minimal modification, we show that MaskCLIP yields compelling segmentation results on open concepts across various datasets in the absence of annotations and fine-tuning. By adding pseudo labeling and self-trai…
Single Domain Generalization (SDG) tackles the problem of training a model on a single source domain so that it generalizes to any unseen target domain. While this has been well studied for image classification, the literature on SDG object detection remains almost non-existent. To address the challenges of simultaneously learning robust object localization and representation, we propose to leverage a pre-trained vision-language model to introduce semantic domain concepts via textual prompts. We achieve this via a semantic augmentation strategy acting on the features extracted by the detector …
The Chinese University of Hong Kong（香港中文大学）；SenseTime Research（商汤科技研究院）；Shanghai Jiao Tong University（上海交通大学）；Qing Yuan Research Institute（清源研究院）；Centre for Perceptual and Interactive Intelligence（感知与交互智能研究中心）
Open-vocabulary detection (OVD) is an object detection task aiming at detecting objects from novel categories beyond the base categories on which the detector is trained. Recent OVD methods rely on large-scale visual-language pre-trained models, such as CLIP, for recognizing novel objects. We identify the two core obstacles that need to be tackled when incorporating these models into detector training: (1) the distribution mismatch that happens when applying a VL-model trained on whole images to region recognition tasks; (2) the difficulty of localizing objects of unseen classes. To overcome t…
Transferring CLIP's Knowledge into Zero-Shot Point Cloud Semantic Segmentation
Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Didi Chuxing（滴滴出行）
CL-CLIP: CLIP-Based Continual Learning Framework with Cost-Volume Category Decoupling for Object Detection
National College for Excellent Engineers, Beihang University（卓越工程师学院，北京航空航天大学）；AI Research, Qihoo 360（360人工智能研究院，奇虎360）；School of Electronic Information Engineering, Beihang University（电子信息学院，北京航空航天大学）；School of Cyber Science and Technology, Beihang University（网络安全科学与技术学院，北京航空航天大学）；School of Computer Science and Engineering, Beihang University（计算机科学与工程学院，北京航空航天大学）；State Key Laboratory of Media Convergence and Communication, Communication University of China（媒体融合与传播国家重点实验室，中国传媒大学）；Institute of Information Technology, Mathematics and Mechanics, Lobachebsky University（信息技术、数学与力学学院，洛瓦茨基大学）；School of Artificial Intelligence, Beihang University（人工智能学院，北京航空航天大学）
Partial CLIP is Enough: Chimera-Seg for Zero-shot Semantic Segmentation
Graduate School of Informatics, Nagoya University（名古屋大学信息学研究科）；AI Thrust, The Hong Kong University of Science and Technology（香港科学与技术大学人工智能研究组）；INSAIT, Sofia University（索菲亚大学INSAIT）
The University of Tokyo（东京大学）；Google Research（谷歌研究院）
Domain generalization (DG) is a difficult transfer learning problem aiming to learn a generalizable model for unseen domains. Recent foundation models (FMs) are robust to many distribution shifts and, therefore, should substantially improve the performance of DG. In this work, we study generic ways to adopt CLIP, a Visual-Language Foundation Model, for DG problems in image classification. While ERM greatly improves the accuracy with bigger backbones and training datasets using standard DG benchmarks, fine-tuning FMs is not practical in many real-world situations. We propose Domain Prompt Learn…
University of Maryland（马里兰大学）；GE Global Research（通用电气全球研究中心）；Avitas Systems（阿维塔斯系统公司）
Visual Domain Adaptation is a problem of immense importance in computer vision. Previous approaches showcase the inability of even deep neural networks to learn informative representations across domain shift. This problem is more severe for tasks where acquiring hand labeled data is extremely hard and tedious. In this work, we focus on adapting the representations learned by segmentation networks across synthetic and real domains. Contrary to previous approaches that use a simple adversarial objective or superpixel information to aid the process, we propose an approach based on Generative Adv…
MSCI: Addressing CLIP's Inherent Limitations for Compositional Zero-Shot Learning
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Key Laboratory of Social Computing and Cognitive Intelligence (Dalian University of Technology), Ministry of Education, China（社会科学计算与认知智能重点实验室（大连理工大学））；The Hong Kong Polytechnic University（香港理工大学）
MagicHOI: Leveraging 3D Priors for Accurate Hand-object Reconstruction from Short Monocular Video Clips
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；ETH Zürich（苏黎世联邦理工学院）；Max Planck Institute for Intelligent Systems（智能系统马克斯·普朗克研究所）
Zero-shot learning (ZSL) algorithms typically work by exploiting attribute correlations to be able to make predictions in unseen classes. However, these correlations do not remain intact at test time in most practical settings and the resulting change in these correlations lead to adverse effects on zero-shot learning performance. In this paper, we present a new paradigm for ZSL that: (i) utilizes the class-attribute mapping of unseen classes to estimate the change in target distribution (target shift), and (ii) propose a novel technique called grouped Adversarial Learning (gAL) to reduce nega…
Describe, Adapt and Combine: Empowering CLIP Encoders for Open-set 3D Object Retrieval
Huazhong Agricultural University（华中农业大学）；Shenzhen University（深圳大学）；The University of Hong Kong（香港大学）；University of Louisville（路易斯安那大学）；ByteDance（字节跳动）；Huazhong University of Science and Technology（华中科技大学）
CT-CLIP: A Multi-modal Fusion Framework for Robust Apple Leaf Disease Recognition in Complex Environments
College of Information Science and Engineering, Shandong Agricultural University（山东农业大学信息科学与工程学院）；Shandong Provincial Smart Agriculture Characteristic Laboratory（山东省智慧农业特色实验室）；Centre for Vision Speech and Signal Processing (CVSSP), University of Surrey（萨里大学视觉语音与信号处理中心）；School of Mechanical and Electronic Engineering, Shandong Agricultural Engineering College（山东农业工程学院机械电子工程学院）；Australian Institute for Machine Learning, University of Adelaide（阿德莱德大学澳大利亚机器学习研究所）
CAPE: A CLIP-Aware Pointing Ensemble of Complementary Heatmap Cues for Embodied Reference Understanding
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Istanbul Technical University（伊斯坦布尔技术大学）；Carnegie Mellon University（卡内基梅隆大学）；KIT Campus Transfer GmbH (KCT)（KIT校园转移有限责任公司）
GazeCLIP: Gaze-Guided CLIP with Adaptive-Enhanced Fine-Grained Language Prompt for Deepfake Attribution and Detection
Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东省科学院））；Shenzhen University（深圳大学）；National Engineering Laboratory for Big Data System Computing Technology, Shenzhen University（深圳大学大数据系统计算技术国家工程实验室）；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳市人工智能与机器人研究院）；Guangdong Key Laboratory of Intelligent Information Processing, Shenzhen University（深圳大学广东省智能信息处理重点实验室）；Great Bay University（大湾区大学）
Guizhou University, Guiyang, China（贵州大学）；Harbin Institute of Technology, China（哈尔滨工业大学）；The Chinese University of Hong Kong, Shenzhen, China（香港中文大学（深圳））
