arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.30745cs.CV
收藏

Immuno-VLM:通过生成式语义抗体实现大型视觉-语言模型的开放世界可信赖性

Immuno-VLM: Immunizing Large Vision-Language Models via Generative Semantic Antibodies for Open-World Trustworthiness

School of Software Engineering, Huazhong University of Science and Technology(华中科技大学软件学院);Nanyang Technological University, Singapore(新加坡南洋理工大学);Nanjing University(南京大学)

Xiang Fang, Wanlong Fang, Wei Ji

AI总结:

针对大型视觉-语言模型在开放世界部署中因缺乏负面知识而将未知异常高置信度误分类为已知类别的“语义傲慢”问题,提出受生物免疫负选择启发的Immuno-VLM框架,利用大语言模型的生成推理主动产生“语义抗体”(近分布异常文本描述)来约束已知类决策空间,在ImageNet-1K和四个OOD基准上达到新最优。

摘要:

大型视觉-语言模型通过将视觉特征与广泛语义概念对齐,在零样本识别中取得了前所未有的成功。然而,这种语义抽象在开放世界部署中造成了一个关键漏洞:“语义傲慢”——由于缺乏显式的负面知识,模型会将未知异常高置信度地强行拟合到已知类别中。为了解决这个“开放世界可信赖性悖论”,我们提出了 extbf{Immuno-VLM},一个受生物启发的框架,它将 extbf{免疫负选择}的生物学原理适应到高维潜在空间。与依赖被动密度估计或低效像素空间异常生成的传统开放集识别方法不同,Immuno-VLM利用大语言模型的生成推理能力主动“幻想”出“语义抗体”,即近分布异常(例如,相似物、上下文异常)的文本描述,这些描述有效地约束了已知类别的决策空间。在ImageNet-1K和四个具有挑战性的OOD基准上的大量实验表明,Immuno-VLM达到了新的最优水平。

英文摘要:

Large Vision-Language Models have achieved unprecedented success in zero-shot recognition by aligning visual features with broad semantic concepts. However, this semantic abstraction creates a critical vulnerability in open-world deployment: the ``Hubris of Semantics'', where models force-fit unknown anomalies into known categories with high confidence due to the lack of explicit negative knowledge. To address this \textit{Open-World Trustworthiness Paradox}, we propose \textbf{Immuno-VLM}, a bio-inspired framework that adapts the biological principle of \textbf{Immunological Negative Selection} to high-dimensional latent spaces. Departing from traditional Open-Set Recognition methods that rely on passive density estimation or inefficient pixel-space outlier generation, Immuno-VLM leverages the generative reasoning of Large Language Models to actively hallucinate ``Semantic Antibodies'', textual descriptions of near-distribution outliers (e.g., look-alikes, contextual anomalies) that effectively bound the decision space of known classes.Extensive experiments on ImageNet-1K and four challenging OOD benchmarks reveal that Immuno-VLM establishes a new state-of-the-art.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑