arXiv 2510.160362025-10-16cs.CV

IAD-GPT：推进多模态大语言模型在工业异常检测中的视觉知识

IAD-GPT: Advancing Visual Knowledge in Multimodal Large Language Model for Industrial Anomaly Detection

School of Computer Science & Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；School of Computing and Information Technology, Great Bay University（大亚湾大学计算机与信息科技学院）；College of Computer Science, Nankai University（南开大学计算机学院）；School of Artificial Intelligence, Shenzhen University（深圳大学人工智能学院）；Guangdong Provincial Key Laboratory of Intelligent Information Processing, Shenzhen University（广东省智能信息处理重点实验室）；National Engineering Laboratory of Big Data System Computing Technology, Shenzhen University（大数据系统计算技术国家工程实验室）

Zewen Li, Zitong Yu, Qilang Ye, Weicheng Xie, Wei Zhuo, Linlin Shen

备注：: Accepted by IEEE Transactions on Instrumentation and Measurement (TIM)

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出IAD-GPT多模态大模型范式，通过异常提示生成、文本引导增强和多掩码融合提升工业异常检测与分割能力。

摘要：

多模态大语言模型（MLLMs）强大的因果能力，为工业异常检测（IAD）中检测缺陷物体提供了潜力。然而，大多数传统IAD方法缺乏提供多轮人机对话和详细描述的能力，例如物体颜色、异常形状或具体异常类型。同时，基于大型预训练模型的方法尚未充分激发大模型在异常检测任务中的能力。本文探索将丰富的文本语义与来自图像的图像级和像素级信息相结合，并提出IAD-GPT，这是一种基于MLLMs的新型IAD范式。我们采用异常提示生成器（APG）为特定物体生成详细的异常提示。来自大语言模型（LLM）的这些特定提示用于激活预训练视觉-语言模型（即CLIP）的检测和分割功能。为增强MLLMs的视觉定位能力，我们提出文本引导增强器，其中图像特征与正常和异常文本提示交互，以动态选择增强路径，使语言模型能够关注视觉数据的特定方面，增强其准确解释和响应图像中异常的能力。此外，我们设计了多掩码融合模块，将掩码作为专家知识纳入，从而增强LLM对像素级异常的感知。在MVTec-AD和VisA数据集上的大量实验表明，我们在自监督和少样本异常检测与分割任务上取得了最先进性能，例如MVTec-AD和VisA数据集。代码位于https://github.com/LiZeWen1225/IAD-GPT。

英文摘要：

The robust causal capability of Multimodal Large Language Models (MLLMs) hold the potential of detecting defective objects in Industrial Anomaly Detection (IAD). However, most traditional IAD methods lack the ability to provide multi-turn human-machine dialogues and detailed descriptions, such as the color of objects, the shape of an anomaly, or specific types of anomalies. At the same time, methods based on large pre-trained models have not fully stimulated the ability of large models in anomaly detection tasks. In this paper, we explore the combination of rich text semantics with both image-level and pixel-level information from images and propose IAD-GPT, a novel paradigm based on MLLMs for IAD. We employ Abnormal Prompt Generator (APG) to generate detailed anomaly prompts for specific objects. These specific prompts from the large language model (LLM) are used to activate the detection and segmentation functions of the pre-trained visual-language model (i.e., CLIP). To enhance the visual grounding ability of MLLMs, we propose Text-Guided Enhancer, wherein image features interact with normal and abnormal text prompts to dynamically select enhancement pathways, which enables language models to focus on specific aspects of visual data, enhancing their ability to accurately interpret and respond to anomalies within images. Moreover, we design a Multi-Mask Fusion module to incorporate mask as expert knowledge, which enhances the LLM's perception of pixel-level anomalies. Extensive experiments on MVTec-AD and VisA datasets demonstrate our state-of-the-art performance on self-supervised and few-shot anomaly detection and segmentation tasks, such as MVTec-AD and VisA datasets. The codes are available at \href{https://github.com/LiZeWen1225/IAD-GPT}{https://github.com/LiZeWen1225/IAD-GPT}.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。