我们提出了一种用于少样本分布外(OOD)检测的新型视觉-语言提示学习方法。少样本 OOD 检测旨在仅使用少量标记的分布内(ID)图像,检测出训练期间未见过的类别的 OOD 图像。尽管 CoOp 等提示学习方法在少样本 ID 分类中表现出有效性和高效性,但由于文本嵌入中可能存在与 ID 无关的信息,它们在 OOD 检测中仍面临局限。为解决此问题,我们引入了一种名为局部正则化上下文优化(LoCoOp)的新方法,该方法在训练期间利用 CLIP 局部特征的部分作为 OOD 特征来执行 OOD 正则化。CLIP 的局部特征包含大量与 ID 无关的干扰因素(如背景),通过学习将这些干扰因素推离 ID 类别文本嵌入,我们可以消除 ID 类别文本嵌入中的干扰因素,并增强 ID 和 OOD 之间的分离。在大规模 ImageNet OOD 检测基准上的实验证明了我们的 LoCoOp 优于零样本、全监督检测方法和提示学习方法。值得注意的是,即使在单样本设置下——每个类别仅有一个标签,LoCoOp 也优于现有的零样本和全监督检测方法。代码将在 https://github.com/AtsuMiyai/LoCoOp 上提供。
英文摘要:
We present a novel vision-language prompt learning approach for few-shot out-of-distribution (OOD) detection. Few-shot OOD detection aims to detect OOD images from classes that are unseen during training using only a few labeled in-distribution (ID) images. While prompt learning methods such as CoOp have shown effectiveness and efficiency in few-shot ID classification, they still face limitations in OOD detection due to the potential presence of ID-irrelevant information in text embeddings. To address this issue, we introduce a new approach called Local regularized Context Optimization (LoCoOp), which performs OOD regularization that utilizes the portions of CLIP local features as OOD features during training. CLIP's local features have a lot of ID-irrelevant nuisances (e.g., backgrounds), and by learning to push them away from the ID class text embeddings, we can remove the nuisances in the ID class text embeddings and enhance the separation between ID and OOD. Experiments on the large-scale ImageNet OOD detection benchmarks demonstrate the superiority of our LoCoOp over zero-shot, fully supervised detection methods and prompt learning methods. Notably, even in a one-shot setting -- just one label per class, LoCoOp outperforms existing zero-shot and fully supervised detection methods. The code will be available via https://github.com/AtsuMiyai/LoCoOp.
Local-Prompt: Extensible Local Prompts for Few-Shot Out-of-Distribution Detection
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Centre for Artificial Intelligence and Robotics, HKISI-CAS(香港中文大学(深圳)人工智能与机器人中心(HKISI-CAS));Southern University of Science and Technology(南方科技大学)
PromptAD: Learning Prompts with only Normal Samples for Few-Shot Anomaly Detection
East China Normal University(华东师范大学);Chongqing Institute of East China Normal University(华东师范大学重庆研究院);The Navy Military Medical University(海军军医大学);Xiamen University(厦门大学)
PSMP-CLIP: Patch-Prompt SAM and Multi-Semantic Prompting for CLIP-Based Zero-Shot Anomaly Detection
Harbin Engineering University(哈尔滨工程大学);Key Laboratory of Advanced Marine Communication and Information Technology(先进海洋通信与信息技术重点实验室);University of Toronto(多伦多大学);Kanagawa University(神奈川大学)
Zero-Shot and Few-Shot Video Question Answering with Multi-Modal Prompts
Inria(法国国家信息与自动化研究所);Univ Rennes(雷恩大学);CNRS(法国国家科学研究中心);IRISA(法国雷恩信息与自动化系统研究所);Institute of Advanced Research in Artificial Intelligence (IARAI)(高级人工智能研究所(IARAI))
DiffuSAM: Diffusion-Based Prompt-Free SAM2 for Few-Shot and Source-Free Medical Image Segmentation
School of Electrical Engineering, Tel Aviv University, Israel(特拉维夫大学电气工程学院);School of Biomedical Engineering, Tel Aviv University, Israel(特拉维夫大学生物医学工程学院)
University of Science and Technology of China(中国科学技术大学);Center for Research on Intelligent Perception and Computing, NLPR, CASIA(中国科学院自动化研究所模式识别国家重点实验室智能感知与计算研究中心);Nanyang Technological University(南洋理工大学);University of Chinese Academy of Sciences(中国科学院大学)
Taiyuan University of Technology(太原理工大学);University of Science and Technology of China(中国科学技术大学);Second Hospital of Shanxi Medical University(山西医科大学第二医院)
Prompting Decision Transformer for Few-Shot Policy Generalization
Carnegie Mellon University(卡内基梅隆大学);University of Montreal(蒙特利尔大学);Mila(米拉);MIT-IBM Watson AI Lab(MIT-IBM沃森人工智能实验室);Massachusetts Institute of Technology(麻省理工学院);UMass Amherst(马萨诸塞大学阿默斯特分校)
Meta-Prompting for Automating Zero-shot Visual Recognition with LLMs
ICG, TU Graz(格拉茨工业大学ICG研究所);CDL-EML(CDL-EML中心);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室);JKU(约翰内斯开普勒大学);IBM Research(IBM研究院);Weizmann Institute of Science(魏茨曼科学研究所);University of Bonn(波恩大学)