This study proposes a novel few-shot crack image classification model based on CLIP and Bayesian optimization. By combining multimodal information and Bayesian approach, the model achieves efficient classification of crack images in a small number of training samples. The CLIP model employs its robust feature extraction capabilities to facilitate precise classification with a limited number of samples. In contrast, Bayesian optimisation enhances the robustness and generalization of the model, while reducing the reliance on extensive labelled data. The results demonstrate that the model exhibits robust performance across a diverse range of dataset scales, particularly in the context of small sample sets. The study validates the potential of the method in civil engineering crack classification.
PSMP-CLIP: Patch-Prompt SAM and Multi-Semantic Prompting for CLIP-Based Zero-Shot Anomaly Detection
Harbin Engineering University（哈尔滨工程大学）；Key Laboratory of Advanced Marine Communication and Information Technology（先进海洋通信与信息技术重点实验室）；University of Toronto（多伦多大学）；Kanagawa University（神奈川大学）
CLIP3D-AD: Extending CLIP for 3D Few-Shot Anomaly Detection with Multi-View Images Generation
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence and Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（西安交通大学人机混合增强智能全国重点实验室与人工智能与机器人研究所）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东省人工智能与数字经济实验室（深圳））；Xiamen University（厦门大学）；Shenzhen University（深圳大学）
AF-CLIP: Zero-Shot Anomaly Detection via Anomaly-Focused CLIP Adaptation
School of Computer Science and Engineering（计算机科学与工程学院）；Sun Yat-sen University（中山大学）；Guangdong Key Laboratory of Big Data Analysis and Processing（大数据分析与处理广东省重点实验室）
MadCLIP: Few-shot Medical Anomaly Detection with CLIP
Department of Computer Science, University of Verona（威尼斯大学计算机科学系）；AI for Good (AIGO) Research Unit, Istituto Italiano di Tecnologia（意大利技术研究院人工智能为善研究单元）
GS-CLIP: Zero-shot 3D Anomaly Detection by Geometry-Aware Prompt and Synergistic View Representation Learning
School of Computer Science and Technology, Soochow University（苏州大学计算机科学与技术学院）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）
AA-CLIP: Enhancing Zero-shot Anomaly Detection via Anomaly-Aware CLIP
School of Biomedical Engineering, Division of Life Sciences and Medicine, USTC（中国科学技术大学生命科学与医学部生物医学工程学院）；MIRACLE Center, Suzhou Institute for Advance Research, USTC（中国科学技术大学苏州高等研究院MIRACLE中心）；Key Laboratory of Intelligent Information Processing of CAS, ICT, CAS（中国科学院计算技术研究所中国科学院智能信息处理重点实验室）；State Key Laboratory of Precision and Intelligent Chemistry, USTC（中国科学技术大学精准与智能化学国家重点实验室）；Stanford University（斯坦福大学）
The application of zero-shot learning in computer vision has been revolutionized by the use of image-text matching models. The most notable example, CLIP, has been widely used for both zero-shot classification and guiding generative models with a text prompt. However, the zero-shot use of CLIP is unstable with respect to the phrasing of the input text, making it necessary to carefully engineer the prompts used. We find that this instability stems from a selective similarity score, which is based only on a subset of the semantically meaningful input tokens. To mitigate it, we present a novel ex…
SPECIAL: Zero-shot Hyperspectral Image Classification With CLIP
School of Mathematics and Statistics and the Ministry of Education Key Laboratory of Intelligent Networks and Network Security, Xi’an Jiaotong University（数学与统计学学院和教育部智能网络与网络安全重点实验室，西安交通大学）；Aerospace Information Research Institute, Chinese Academy of Sciences（航天信息研究所，中国科学院）；School of Computer Science and Technology and the Ministry of Education Key Laboratory for Intelligent Networks and Network Security, Xi’an Jiaotong University（计算机科学与技术学院和教育部智能网络与网络安全重点实验室，西安交通大学）；School of Information and Communication Technology, Griffith University（信息与通信技术学院，格里菲斯大学）
CLIP (Contrastive Language-Image Pre-training) models developed by OpenAI have achieved outstanding results on various image recognition and retrieval tasks, displaying strong zero-shot performance. This means that they are able to perform effectively on tasks for which they have not been explicitly trained. Inspired by the success of OpenAI CLIP, a new publicly available dataset called LAION-5B was collected which resulted in the development of open ViT-H/14, ViT-G/14 models that outperform the OpenAI L/14 model. The LAION-5B dataset also released an approximate nearest neighbor index, with a…
Enhancing Zero-Shot Anomaly Detection: CLIP-SAM Collaboration with Cascaded Prompts
School of Artificial Intelligence, Anhui University, Hefei, China（人工智能学院，安徽大学，合肥，中国）；Anhui Provincial Key Laboratory of Security Artificial Intelligence, Anhui University（安徽省安全人工智能重点实验室，安徽大学）；Anhui Provincial Engineering Research Center for Unmanned System（安徽省无人系统与智能技术工程研究中心）
本文提出一种用于工业异常检测的两阶段零样本异常分割框架，协同利用 CLIP 的异常定位与 SAM 的边界感知能力，通过 PPG 和 CPS 模块优化点提示与分割精度，在 Visa 等数据集上取得 SOTA 表现。
CLIP-FSAC++: Few-Shot Anomaly Classification with Anomaly Descriptor Based on CLIP
Xi’an Jiaotong University（西安交通大学）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东省人工智能与数字经济实验室（深圳））；Xiamen University（厦门大学）；Shenzhen University（深圳大学）
National Technical University of Athens（雅典国立技术大学）；University of Macedonia（马其顿大学）
Recent studies indicate that deep learning plays a crucial role in the automated visual inspection of road infrastructures. However, current learning schemes are static, implying no dynamic adaptation to users' feedback. To address this drawback, we present a few-shot learning paradigm for the automated segmentation of road cracks, which is based on a U-Net architecture with recurrent residual and attention modules (R2AU-Net). The retraining strategy dynamically fine-tunes the weights of the U-Net as a few new rectified samples are being fed into the classifier. Extensive experiments show that…
Label Shift Aware Adaptation for Online Zero-shot Learning with Contrastive Language-Image Pre-Training (CLIP)
Australian National University（澳大利亚国立大学）；China North Vehicle Research Institute（中国北方车辆研究所）；The Hong Kong Polytechnic University（香港理工大学）；Griffith University（格里菲斯大学）；CSIRO（澳大利亚联邦科学与工业研究组织）；The University of Hong Kong（香港大学）
Istituto di Matematica Applicata e Tecnologie Informatiche ‘E. Magenes’ - CNR（意大利国家研究委员会“E. 马涅斯”应用数学与信息技术研究所）；University of Chile（智利大学）；University of Science, Ho Chi Minh City（胡志明市科学大学）；Viet Nam National University, Ho Chi Minh City（胡志明市越南国家大学）；John von Neumann Institute（约翰·冯·诺依曼研究所）
This paper describes the methods submitted for evaluation to the SHREC 2022 track on pothole and crack detection in the road pavement. A total of 7 different runs for the semantic segmentation of the road surface are compared, 6 from the participants plus a baseline method. All methods exploit Deep Learning techniques and their performance is tested using the same environment (i.e.: a single Jupyter notebook). A training set, composed of 3836 semantic segmentation image/mask pairs and 797 RGB-D video clips collected with the latest depth cameras was made available to the participants. The meth…
Harbin Institute of Technology（哈尔滨工业大学）；Microsoft Research（微软研究院）
CLIP has shown a remarkable zero-shot capability on a wide range of vision tasks. Previously, CLIP is only regarded as a powerful visual encoder. However, after being pre-trained by language supervision from a large amount of image-caption pairs, CLIP itself should also have acquired some few-shot abilities for vision-language tasks. In this work, we empirically show that CLIP can be a strong vision-language few-shot learner by leveraging the power of language. We first evaluate CLIP's zero-shot performance on a typical visual question answering task and demonstrate a zero-shot cross-modality …
Explaining CLIP Zero-shot Predictions Through Concepts
School of Informatics, University of Edinburgh（爱丁堡大学信息学院）；Dept. of Computer Eng., Middle East Technical University (METU)（中东技术大学计算机工程系）；Orbital；DTU Compute, Technical University of Denmark（丹麦技术大学DTU计算学院）；Dept. of Biology, University of Copenhagen（哥本哈根大学生物系）；LTCI, Télécom Paris, Institut Polytechnique de Paris（巴黎理工学院巴黎电信学院LTCI实验室）；Technical University of Munich (TUM)（慕尼黑工业大学）；Helmholtz Munich（亥姆霍兹慕尼黑中心）；MCML；MDSI；Robotics & AI Center (ROMER), METU（中东技术大学机器人与人工智能中心）
fine-CLIP: Enhancing Zero-Shot Fine-Grained Surgical Action Recognition with Vision-Language Models
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；INSERM（法国国家健康与医学研究院）；ICube（ICube实验室）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；University Hospital of Strasbourg（斯特拉斯堡大学医院）
