arXiv 2503.233882025-03-30cs.CVcs.AIcs.LGcs.MM

COSMIC：面向团族的语义多空间集成用于鲁棒的 CLIP 测试时适应

COSMIC: Clique-Oriented Semantic Multi-space Integration for Robust CLIP Test-Time Adaptation

Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Shenzhen Technology University（深圳技术大学）；University of Science and Technology of China（中国科学技术大学）

Fanding Huang, Jingyan Jiang, Qinting Jiang, Hebei Li, Faisal Nadeem Khan, Zhi Wang

备注：: Accepted to CVPR 2025

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出COSMIC测试时适应框架，通过引入Dual Semantics Graph和Clique Guided Hyper-class构建多粒度跨模态语义缓存与图查询机制，有效提升了CLIP在分布外和跨域任务上的适应准确率。

摘要：

近期的视觉语言模型（VLMs）在新领域的测试时适应中面临重大挑战。尽管基于缓存的方法通过利用历史信息展现出潜力，但它们在缓存不可靠的特征-标签对以及在查询时无差别地使用单类信息方面存在困难，这显著影响了适应准确性。为了解决这些局限性，我们提出了 COSMIC（面向团族的语义多空间集成用于 CLIP），这是一个鲁棒的测试时适应框架，通过多粒度、跨模态语义缓存和基于图的查询机制来增强适应性。我们的框架引入了两个关键创新：Dual Semantics Graph (DSG) 和 Clique Guided Hyper-class (CGH)。Dual Semantics Graph 通过结合文本特征、粗粒度 CLIP 特征和细粒度 DINOv2 特征来构建互补的语义空间，以捕获丰富的语义关系。在这些双图的基础上，Clique Guided Hyper-class 组件利用结构化的类关系，通过相关类选择来增强预测鲁棒性。大量实验表明，COSMIC 在多个基准上表现出卓越的性能，相较于最先进的方法取得了显著提升：在 CLIP RN-50 上，分布外任务提升了 15.81%，跨域生成提升了 5.33%。代码可在 github.com/hf618/COSMIC 获取。

英文摘要：

Recent vision-language models (VLMs) face significant challenges in test-time adaptation to novel domains. While cache-based methods show promise by leveraging historical information, they struggle with both caching unreliable feature-label pairs and indiscriminately using single-class information during querying, significantly compromising adaptation accuracy. To address these limitations, we propose COSMIC (Clique-Oriented Semantic Multi-space Integration for CLIP), a robust test-time adaptation framework that enhances adaptability through multi-granular, cross-modal semantic caching and graph-based querying mechanisms. Our framework introduces two key innovations: Dual Semantics Graph (DSG) and Clique Guided Hyper-class (CGH). The Dual Semantics Graph constructs complementary semantic spaces by incorporating textual features, coarse-grained CLIP features, and fine-grained DINOv2 features to capture rich semantic relationships. Building upon these dual graphs, the Clique Guided Hyper-class component leverages structured class relationships to enhance prediction robustness through correlated class selection. Extensive experiments demonstrate COSMIC's superior performance across multiple benchmarks, achieving significant improvements over state-of-the-art methods: 15.81% gain on out-of-distribution tasks and 5.33% on cross-domain generation with CLIP RN-50. Code is available at github.com/hf618/COSMIC.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。