Recently, zero-shot and few-shot learning via Contrastive Vision-Language Pre-training (CLIP) have shown inspirational performance on 2D visual recognition, which learns to match images with their corresponding texts in open-vocabulary settings. However, it remains under explored that whether CLIP, pre-trained by large-scale image-text pairs in 2D, can be generalized to 3D recognition. In this paper, we identify such a setting is feasible by proposing PointCLIP, which conducts alignment between CLIP-encoded point cloud and 3D category texts. Specifically, we encode a point cloud by projecting it into multi-view depth maps without rendering, and aggregate the view-wise zero-shot prediction to achieve knowledge transfer from 2D to 3D. On top of that, we design an inter-view adapter to better extract the global feature and adaptively fuse the few-shot knowledge learned from 3D into CLIP pre-trained in 2D. By just fine-tuning the lightweight adapter in the few-shot settings, the performance of PointCLIP could be largely improved. In addition, we observe the complementary property between PointCLIP and classical 3D-supervised networks. By simple ensembling, PointCLIP boosts baseline's performance and even surpasses state-of-the-art models. Therefore, PointCLIP is a promising alternative for effective 3D point cloud understanding via CLIP under low resource cost and data regime. We conduct thorough experiments on widely-adopted ModelNet10, ModelNet40 and the challenging ScanObjectNN to demonstrate the effectiveness of PointCLIP. The code is released at https://github.com/ZrrSkywalker/PointCLIP.
CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-training
Harbin Institute of Technology（哈尔滨工业大学）；City University of Hong Kong（香港城市大学）；The University of Hong Kong（香港大学）；Shanghai AI Laboratory（上海人工智能实验室）；Peng Cheng Laboratory（鹏城实验室）
PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learning
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；City University of Hong Kong（香港城市大学）；Yale University（耶鲁大学）；Peking University（北京大学）；The Chinese University of Hong Kong（香港中文大学）
Position-aware Guided Point Cloud Completion with CLIP Model
North China University of Technology（北方工业大学）；Peng Cheng Laboratory（鹏城实验室）；University of Copenhagen（哥本哈根大学）；University of Washington（华盛顿大学）；Skywork AI（天工智能）；Tsinghua University（清华大学）
CLIP-GS: Unifying Vision-Language Representation with 3D Gaussian Splatting
Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学学院）；National University of Singapore（新加坡国立大学）；Meituan（美团）；Georgia Institute of Technology（佐治亚理工学院）；Picsart AI Research (PAIR)（Picsart AI研究（PAIR））
Contrastive Language-Image Pre-training, benefiting from large-scale unlabeled text-image pairs, has demonstrated great performance in open-world vision understanding tasks. However, due to the limited Text-3D data pairs, adapting the success of 2D Vision-Language Models (VLM) to the 3D space remains an open problem. Existing works that leverage VLM for 3D understanding generally resort to constructing intermediate 2D representations for the 3D data, but at the cost of losing 3D geometry information. To take a step toward open-world 3D vision understanding, we propose Contrastive Language-Imag…
Fine-grained Text and Image Guided Point Cloud Completion with CLIP Model
Dalian Maritime University（大连海事大学）；Zhejiang Sci-Tech University（浙江理工大学）；Beijing University of Technology（北京工业大学）；Dalian University of Technology（大连理工大学）
Peking University（北京大学）；Shanghai AI Laboratory（上海人工智能实验室）；ShanghaiTech University（上海科技大学）
Contrastive Language-Image Pre-training (CLIP) has been shown to learn visual representations with great transferability, which achieves promising accuracy for zero-shot classification. To further improve its downstream performance, existing works propose additional learnable modules upon CLIP and fine-tune them by few-shot training sets. However, the resulting extra training cost and data requirement severely hinder the efficiency for model deployment and knowledge transfer. In this paper, we introduce a free-lunch enhancement method, CALIP, to boost CLIP's zero-shot performance via a paramet…
Describe, Adapt and Combine: Empowering CLIP Encoders for Open-set 3D Object Retrieval
Huazhong Agricultural University（华中农业大学）；Shenzhen University（深圳大学）；The University of Hong Kong（香港大学）；University of Louisville（路易斯安那大学）；ByteDance（字节跳动）；Huazhong University of Science and Technology（华中科技大学）
UniPart: Towards Zero-shot Language-Grounded 3D Part Segmentation for Embodied Interaction
NLPR, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别国家重点实验室）；Peking University（北京大学）；Tsinghua University（清华大学）；Galbot Inc.（北京银河通用机器人有限公司）；Shanghai Jiao Tong University（上海交通大学）；Beijing Academy of Artificial Intelligence（北京智源人工智能研究院）
More Text, Less Point: Towards 3D Data-Efficient Point-Language Understanding
Huazhong University of Science and Technology（华中科技大学）；Guangdong Intelligent Robotics Institute（广东省智能机器人研究院）；Guangdong HUST Industrial Technology Research Institute（广东华中科技大学工业技术研究院）；South China University of Technology（华南理工大学）；Pazhou Laboratory（琶洲实验室）
OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understanding
School of Electronic and Computer Engineering, Peking University（北京大学电子与计算机工程学院）；Peking University Shenzhen Graduate School（北京大学深圳研究生院）；Baidu VIS（百度视觉智能部）；Beihang University（北京航空航天大学）
Transferring CLIP's Knowledge into Zero-Shot Point Cloud Semantic Segmentation
Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Didi Chuxing（滴滴出行）
VoteSplat: Hough Voting Gaussian Splatting for 3D Scene Understanding
School of Computer Science and Technology, Xidian University（西安电子科技大学计算机科学与技术学院）；Algorithm R&D Center, Qing Yi (Shanghai)（清一（上海）算法研发中心）；Shanghai Pudong Cryptography Research Institute（上海浦东密码研究院）；Division of Decision and Control Systems and Department of Mathematics, KTH Royal Institute of Technology（皇家理工学院决策与控制系统系和数学系）
Text-guided Feature Disentanglement for Cross-modal Gait Recognition
Fujian Key Laboratory of Urban Intelligent Sensing and Computing, Xiamen University（福建城市智能感知与计算重点实验室，厦门大学）；Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（多媒体可信感知与高效计算重点实验室，中华人民共和国教育部，厦门大学）
CLIP3D-AD: Extending CLIP for 3D Few-Shot Anomaly Detection with Multi-View Images Generation
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence and Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（西安交通大学人机混合增强智能全国重点实验室与人工智能与机器人研究所）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东省人工智能与数字经济实验室（深圳））；Xiamen University（厦门大学）；Shenzhen University（深圳大学）
Recent advancements in vision-language pre-training (e.g. CLIP) have shown that vision models can benefit from language supervision. While many models using language modality have achieved great success on 2D vision tasks, the joint representation learning of 3D point cloud with text remains under-explored due to the difficulty of 3D-Text data pair acquisition and the irregularity of 3D data structure. In this paper, we propose a novel Text4Point framework to construct language-guided 3D point cloud models. The key idea is utilizing 2D images as a bridge to connect the point cloud and the lang…
Parameter-Efficient CLIP Adaptation for 3D Understanding via Unified Tokenization
Fondazione Bruno Kessler（布鲁诺·科塞拉基金会）；University of Trento（特伦托大学）；University of Pisa（比萨大学）；Beijing Forestry University（北京林业大学）；Shanghai Jiao Tong University（上海交通大学）；Hong Kong University of Science and Technology (GZ)（香港科技大学）；Shandong University（山东大学）；University of California, Merced（加州大学默塞德分校）
CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIP
The University of Hong Kong（香港大学）；Shanghai AI Laboratory（上海人工智能实验室）；Hochschule Bremerhaven（不来梅哈芬应用技术大学）；National University of Singapore（新加坡国立大学）；The Chinese University of Hong Kong（香港中文大学）；ShanghaiTech University（上海科技大学）；Texas A&M University（德州农工大学）
Training a 3D scene understanding model requires complicated human annotations, which are laborious to collect and result in a model only encoding close-set object semantics. In contrast, vision-language pre-training models (e.g., CLIP) have shown remarkable open-world reasoning properties. To this end, we propose directly transferring CLIP's feature space to 3D scene understanding model without any form of supervision. We first modify CLIP's input and forwarding process so that it can be adapted to extract dense pixel features for 3D scene contents. We then project multi-view image features t…
CLIPose: Category-Level Object Pose Estimation with Pre-trained Vision-Language Knowledge
College of Electronics and Information Engineering, Tongji University（同济大学电子与信息工程学院）；Department of Electrical and Computer Engineering, Wayne State University（韦恩州立大学电气与计算机工程学院）
GS-CLIP: Zero-shot 3D Anomaly Detection by Geometry-Aware Prompt and Synergistic View Representation Learning
School of Computer Science and Technology, Soochow University（苏州大学计算机科学与技术学院）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）
University of Rochester（罗切斯特大学）；Nanyang Technological University（南洋理工大学）；Shanghai AI Laboratory（上海人工智能实验室）
The recent success of pre-trained 2D vision models is mostly attributable to learning from large-scale datasets. However, compared with 2D image datasets, the current pre-training data of 3D point cloud is limited. To overcome this limitation, we propose a knowledge distillation method for 3D point cloud pre-trained models to acquire knowledge directly from the 2D representation learning model, particularly the image encoder of CLIP, through concept alignment. Specifically, we introduce a cross-attention mechanism to extract concept features from 3D point cloud and compare them with the semant…
