Triad: 通过视觉专家引导的视觉标记器和制造过程增强基于LMM的异常检测
Triad: Empowering LMM-based Anomaly Detection with Vision Expert-guided Visual Tokenizer and Manufacturing Process
AI总结：
Triad通过引入视觉专家引导的标记器和制造过程，提升基于LMM的工业异常检测性能。
摘要：
尽管最近的方法尝试将大型多模态模型（LMMs）引入工业异常检测（IAD），但其在IAD领域的泛化能力远逊于通用用途。我们总结出这一差距的主要原因分为两个方面。一方面，通用的LMMs在视觉模态中缺乏对缺陷的认知，因此无法充分聚焦于缺陷区域。因此，我们提出修改LLaVA模型的AnyRes结构，将现有IAD模型所识别的潜在异常区域提供给LMMs。另一方面，现有方法主要通过学习缺陷模式或与正常样本进行比较来识别缺陷，但未能理解这些缺陷的原因。考虑到缺陷的产生与制造过程密切相关，我们提出了一种以制造过程为导向的IAD范式。设计了一个用于IAD的指令微调数据集（InstructIAD）和一种用于链式推理的制造（CoT-M）数据组织方法，以利用制造过程进行IAD。基于上述两个修改，我们提出了Triad，一种新的基于LMM的方法，结合了专家引导的感兴趣区域标记器和制造过程用于工业异常检测。广泛的实验表明，我们的Triad不仅在与当前LMMs的竞争中表现出色，而且在配备制造过程时还实现了进一步的准确性提升。源代码、训练数据和预训练模型将在https://github.com/tzjtatata/Triad上公开。
英文摘要：
Although recent methods have tried to introduce large multimodal models (LMMs) into industrial anomaly detection (IAD), their generalization in the IAD field is far inferior to that for general purposes. We summarize the main reasons for this gap into two aspects. On one hand, general-purpose LMMs lack cognition of defects in the visual modality, thereby failing to sufficiently focus on defect areas. Therefore, we propose to modify the AnyRes structure of the LLaVA model, providing the potential anomalous areas identified by existing IAD models to the LMMs. On the other hand, existing methods mainly focus on identifying defects by learning defect patterns or comparing with normal samples, yet they fall short of understanding the causes of these defects. Considering that the generation of defects is closely related to the manufacturing process, we propose a manufacturing-driven IAD paradigm. An instruction-tuning dataset for IAD (InstructIAD) and a data organization approach for Chain-of-Thought with manufacturing (CoT-M) are designed to leverage the manufacturing process for IAD. Based on the above two modifications, we present Triad, a novel LMM-based method incorporating an expert-guided region-of-interest tokenizer and manufacturing process for industrial anomaly detection. Extensive experiments show that our Triad not only demonstrates competitive performance against current LMMs but also achieves further improved accuracy when equipped with manufacturing processes. Source code, training data, and pre-trained models will be publicly available at https://github.com/tzjtatata/Triad.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
